Method and apparatus for video data compression using temporally adaptive motion interpolation

ABSTRACT

In a system for compressing video data, temporally adaptive motion interpolation based upon temporal characteristics of human vision is used for establishing threshold levels relative to the degree of motion as a whole (global motion) between frames. The global motion between successive frames in a group of pictures (GOP) is measured to determine if the motion is less or greater than the established threshold levels for determining the designation of I, P, and B frames, spacing between I and P reference frames, and the number of bits used for each frame, and B frames therebetween.

FIELD OF THE INVENTION

The field of the present invention relates generally to digital datacompression, and more particularly relates to the coding of successiveframes of video data in a selective manner providing enhanced datacompression.

BACKGROUND OF THE INVENTION

The availability of high speed digital devices and large, fast memorieshas made it possible to give practiced expression to an old idea of moreefficiently utilizing a given bandwidth for a video transmission by onlytransmitting encoded digital signals representing the changes betweensuccessive frames, or groups of frames. To achieve high compression, onemust resort not only to redundancy reduction but also to irrelevancyreduction, coarse coding that exploits characteristics of human visualperception. Spatial limits in human vision have been exploitedextensively in many systems, especially in adaptive quantization usingthe discrete cosine transform (e.g., in the DCT quantization matrix),and in other techniques such as subband coding, and multiresolutionrepresentation. Temporal data reduction is based upon the recognitionthat between successive frames of video images there is highcorrelation. However, there has been very little work on applyingtemporal characteristics of human vision to image coding systems, exceptin the most basic ways such as determining a frame rate, e.g. 24-60frames/secs. This is partly because of the anticipated higher complexityof temporal processing than of spatial processing, and the difficulty ofincluding the temporal dimension in defining a standard measure ofperceptual quality for video sequences.

In a standard promulgated in November 1991 by the Motion Picture ExpertGroup, MPEG identified as (ISO-IEC/JTC1/SC2/WG12), the sequence of rawimage data frames are divided into successive groups known as GOP's(group of pictures), respectively, and the coded GOP is comprised ofindependent frames I, predicted frames P and bidirectionally predictedframes B in such manner that GOP may be comprised, for example, asfollows:

I, B, B, P, B, B, P, B, B, P, B, B, P, B, B.

The first P frame is derived from the previous I frame, while theremaining P frames are derived from the last previous P frame. The I andP frames are reference frames. Since each B frame is derived from theclosest reference frames on either side of it, the pertinent P framesmust be derived before the prior B frames can be derived.

The high definition independent frames I at the beginning of each GOPare required because of the use of frame differential encoding to avoidaccumulated error. The purpose of quantizing is to control the number ofbits used in representing the changes between the frames. Thecorresponding portions of frames are conveyed by motion vectorsindicating where blocks in the reference frames may be located in theframe being derived. Since differences generally arise from motion andthere is likely to be more motion between P frames than between a P anda B frame, more bits are required to derive a P frame from a P framethan in deriving B frame from P frames on either side of it.

In typical MPEG systems, the three frame spacing of reference framesutilizing high numbers of bits is required in order to adequately conveymotion. If, however, there is little or no motion, the number of bitsdevoted to representing these frames is excessive.

SUMMARY OF THE INVENTION

One object of the invention is to provide an improved method andapparatus for video compression.

Another object of the invention is to provide an improved method andapparatus for motion compensated interpolation coding for videocompression, which is compatible with present video coding standards.

In one embodiment of the invention, temporal segmentation is used todynamically adapt motion interpolation structures for video compression.Temporal variation of the input video signal is used for adjusting theinterval between reference frames. The bit rate control for the dynamicgroup of pictures (GOP) structure is based upon the temporal masking inhuman vision. In a preferred embodiment, the spacing between referenceframes is based upon different temporal distance measurements, throughuse of an algorithm for temporally adaptive motion interpolation (TAMI).These and other embodiments of the invention are summarized in greaterdetail in the following paragraphs.

Instead of rigidly locating reference frames within a GOP as in aconventional MPEG system, their location and the number of bits useddepends in this invention on the amount of global motion between frames.Global motion as used in the invention, is defined as the motion betweenframes as a whole, and it can be measured in a number of known ways suchas the difference of histogram, the histogram of difference, blockhistogram difference, block variance difference, and motion compensationerror.

In the following description, frames corresponding to I, P, and B framesof the MPEG are designated as I1, P1, and B1 frames.

If there is no frame in a GOP where the global motion between it and theI1 or first frame exceeds an empirically determined value, T₀, all ofthe remaining frames are of the B Type and are derived from the I1 frameof that GOP and the I1 of the next GOP. Thus no P1 type frame is usedand many bits are saved, in this embodiment of the invention.

Should the global motion between a frame and a previous reference frameexceed T₀, the previous frame is designated herein as a P1 frame. Thus,a P1 frame is used wherever necessary to adequately convey the motion.

The global motion between adjacent frames is also measured. If itexceeds an empirically determined value of T₁, it is an indication thatan abrupt change of scene has occurred between the frames. In this casethe later frame is designated as an I2 frame that is independentlyprocessed with fewer bits than the I1 frame, and the immediatelyprevious frame is designated as a P2 frame that has fewer bits than a P1frame in this embodiment of the invention. The relative coarseness ofthe P2 frame is not observed by the eye because of a phenomenon known asbackward masking, and the relative coarseness of the I2 frame is notobserved by the eye because of a phenomenon known as forward masking.

It is apparent in both a conventional MPEG system and the system of thisinvention that reference frames must be processed before the B or B1frames between them can be processed.

The method of operation just described uses bits only as required byglobal motion and is referred to infra as temporally adaptive motioninterpolation, TAMI.

When the system is used with a transmission channel such as used indigital television, the bit rate may be controlled by loading theprocessed bits into a buffer and controlling the number of levels usedin the quantizer so as to keep a given number of bits in the buffer. Inthe event that two or more successive frames have global motion inexcess of T₀ so that the frames just prior to them are designated asgood resolution P1 frames, it is possible that controlling the bit ratemay cause a second P1 frame to be processed with fewer bits thandesired. In such a case, only the first P1 frame is processed, and theframes between it and the next reference frames are processed as B1frames even though they may qualify as P1 frames, in another embodimentof the invention.

Another way of controlling the quantizer so as not to exceed a fixed bitrate is to look at the total number of bits called for in a GOP if thenominal numbers of bits are used for processing the designated frames,and if it calls for a bit rate in excess of the fixed bit rate, thenominal numbers of bits are lowered proportionately as required. Thus,if there are too many P1 frames, the quantized levels are reduced sothat fewer bits are used in processing all of them, in anotherembodiment of the invention.

If the coding system of this invention is coupled to a distributionsystem, such as one using the asynchronous transfer mode (ATM), conceptof a broadband integrated services digital network (ISDN), in anotherembodiment variable bit rate (VBR) coding can be used with TAMI to forma VBR-TAMI system because of the very wide effective bandwidth of such achannel. This system is different from TAMI only in the fact that thenumber of P1 frames is not limited.

In fixed bit rate TAMI (FBR-TAMI), there is as in any block motioncompensation coding system, a tendency for reference frames to be toofar apart e.g. when there is no global motion in excess of T₀, so as toproduce perceptually displeasing coding errors at moving edges.

Furthermore, the longest encoding delay in FBR-TAMI or VBR-TAMI is equalto the duration of a GOP, which in the case of a GOP of fifteen framesis one half a second, and thus may be too long.

In order to alleviate these problems, N-P1 reference frames are insertedinto the GOP structure by default i.e. they are not called for by theglobal motion involved, in yet another embodiment of the invention. Thisdivides the GOP into N+1 segments that may be processed in sequences soas to reduce the time between reference frames and provide better motionrepresentation as well as to reduce the processing time. As N increasesthe coding delay is reduced but so are the bit rate savings.

If no frame in a GOP is designated as a P1 frame because global motionfrom I1 did not exceed T₀, a P1 frame is replaced by a B1 frame so that|P1|-|B1| extra bits are available for processing a B type frame. (Theabsolute value notation is used to denote the number of bits allocatedto a frame.) A B Type frame with the extra bits is called a B2 frame, inthis embodiment of the invention. The relation between |B1| and |B2| isgiven by the following expression: ##EQU1## where M is the GOP size andN is the number of injected P1 frames.

After the frames of a GOP have been designated as P1, and B1 or B2frames in any system or embodiment of the invention, the calculationsrequired for the interpolations of the B1, or B2 frames may beaccomplished by the usual motion compensation encoder, but it ispreferable to use an encoder that uses telescopic motion vectorsearching as well as a different bit rate control system.

In a preferred embodiment of the invention P1 frames are located so asto have as close to the same amount of motion between them as possiblerather than being located at a frame having a motion with respect to theprevious reference frame that exceeds T₀. Differences in motion for eachframe are generated, and as before, I2 and P2 frames are designated at ascene change when the motion between them exceeds T₁. A number N of P1frames to be used is assumed, and calculations are made of the temporaldistances between them and the reference frames on either side for allcombinations of N positions, and the positions closest to the mean ofall these measurements are selected. This embodiment of the invention isdesignated as OSA, for optimal spacing algorithm.

In another embodiment of the invention the most advantageous number, N,of P frames is determined on a dynamic basis from the frame designationsin each GOP.

Another embodiment of the invention applies TAMI to subband videocoding. In view of the fact that highly accurate motion vectorinformation is not required in carrying out the algorithms associatedwith the various embodiments of the invention, the number ofcomputations can be reduced by using pixel subsampling in both spatialdirections and more reduction in calculations can be achieved by usingbackward telescopic searches rather than forward telescopic searches.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the present invention are described below withreference to the drawings, in which substantially similar components areidentified by the same reference designation, wherein:

FIG. 1 shows equivalent intensity pulses illustrating luminanceperception by the human eye under Bloch's law.

FIG. 2A shows a plot of the frequency response function of a first orderlow pass filter model of the temporal frequency response of the humaneye.

FIG. 2B is a curve showing the contrast sensitivity versus the temporalfrequency for a typical temporal modulation transfer function.

FIG. 3 shows a block diagram of a video encoder providing temporallyadaptive motion interpolation (TAMI) of a group of pictures (GOP), forone embodiment of the invention.

FIG. 4 shows a flowchart for the TAMI programming or algorithmassociated with the encoder of FIG. 3, for one embodiment of theinvention.

FIG. 5 shows plots or curves of signal-to-noise ratio (SNR) versus framenumber for comparing 0-P processing when the number of P1 frames is notlimited, versus the signal-to-noise ratio obtained when the number of P1frames is limited, in one embodiment of the invention.

FIG. 6 illustrates a telescopic search for motion estimation betweensuccessive frames as used in one embodiment of the invention.

FIGS. 7A through 7D show default group of picture (GOP) structures for NP1 frames, for N=0, 1, 2, and 3, respectively, as used in embodiments ofthe invention.

FIG. 8A illustrates a GOP structure for a 1-P scheme with scene changesof Type 1 and Type 0, in one embodiment of the invention.

FIG. 8B illustrates a GOP structure for a 1-P scheme with a scene changeof Type 1 only, and no detection of Type 0 scene changes.

FIG. 9 shows a variable bit rate TAMI encoder for another embodiment ofthe invention.

FIG. 10 shows a flowchart of a variable bit rate TAMI algorithmassociated with the encoder of FIG. 9.

FIG. 11 shows a flowchart for providing an optimal spacing algorithm foranother embodiment of the invention.

FIG. 12 shows a GOP structure using the 2-P optimal spacing algorithm ofFIG. 11, when there is a scene change of Type 1.

FIG. 13 illustrates a backward telescopic search for use with theoptimal spacing algorithms of various embodiments of the invention.

FIGS. 14(a) through 14(e) show signal-to-noise ratio (SNR) curves forimages with little motion when the average bit-rate is 736.5Kbit/sec(Tennis), for comparing a conventional fixed 4-P scheme, and 0-P, 1-P,2-P, and 3-P schemes, respectively.

FIGS. 15(a) through 15(e) are related to the SNR curves of FIGS. 14(a)through 14(e), respectively, for showing the corresponding bit rate perframe, where the average bit rate is 736.5Kbit/sec (Tennis).

FIGS. 16(a) through 16(e) show SNR curves derived from a plurality ofsuccessive frames of a GOP with a scene change having an average bitrate of 736.5Kbit/sec (Tennis), for conventional fixed 4-P, and 0-P,1-P, 2-P, and 3-P schemes, respectively, for an embodiment of theinvention.

FIGS. 17(a) through 17(e) show curves for the bit rate versus successiveframe numbers for high temporal activity regions with an abrupt scenechange, when the average bit rate is 736.5Kbit/sec (Tennis),corresponding to FIGS. 17(a) through 17(e), respectively.

FIGS. 18(a) through 18(e) show SNR curves versus successive framenumbers for images with little motion when the bit rate is 300Kbit/sec(Tennis), for conventional fixed 4-P, and for inventive embodiments for0-P, 1-P, 2-P, and 3-P schemes, respectively.

FIGS. 19(a) through 19(e) show the bit rates versus successive framenumbers corresponding to FIGS. 18(a) through 18(e), respectively.

FIG. 20 is a table showing the performance of different interpolationschemes for images with little motion activity at average bit rates of736.5Kbit/sec, for conventional fixed 4-P, and inventive 0-P, 1-P, 2-P,and 3-P embodiments, respectively.

FIG. 21 is a table for showing the performance of differentinterpolation schemes for images containing a scene change at an averagebit rate of 736.5Kbit/sec.

FIG. 22 shows a table for illustrating the performance of differentinterpolation schemes for images with little motion activity at anaverage bit rate of 300Kbit/sec.

FIG. 23(a) shows SNR curves for comparing the performance of theFBR-TAMI and VBR-TAMI embodiments of the invention, respectively, havingthe same average bit rate of 663Kbit/sec.

FIG. 23(b) shows the bit rates per frame for comparing curves in usingFBR-TAMI, and VBR-TAMI, respectively, with the same average bit rate of663Kbit/sec.

FIGS. 24(a) through 24(e) show the distances between a current frame anda first frame for frame numbers 120 through 180, using DOH, HOD, BH, BV,and MCE measurement methods, respectively.

FIGS. 25(a) through 25(e) show SNR curves of an optimal spacingalgorithm (OSA) of the present invention using DOH, HOD, BH, BV, and MCEmeasurements, respectively.

FIG. 26 shows a table for tabulating the results of SNR and bit-rateresults using the OSA embodiment of the invention without B2 frames forthe five different distance measurement methods, and for three differentframe number ranges.

FIG. 27 shows a table for tabulating SNR and bit-rate results using theOSA embodiment of the invention with B2 frames for the five differentdistance measures, and for three different frame number ranges.

FIGS. 28(a) through 28(e) show curves of SNR versus frame numbers 90through 150 through use of the embodiment of the invention of anadaptive optimal spacing algorithm (OSA) with B2 frames using distancemeasurement methods, DOH, HOD, BH, BV, and MCE, respectively.

FIG. 29 shows a composite of curves for comparing the TAMI and OSAembodiments of the invention.

FIG. 30 shows distances between one frame and others relative totemporal segments using a Type 0 threshold set at τ.

FIG. 31 shows an algorithm for another embodiment of the inventiondesignated BS E-TAMI (Binary search equidistant TAMI).

FIG. 32 shows a flowchart relative to both TAMI and OSA embodiments ofthe invention.

FIG. 33 shows a flowchart for the steps involved in a scene changedetection step generally called for in the flowchart of FIG. 32.

FIGS. 34A and 34B show another flowchart for a scene change detectionmethod for another embodiment of the invention designated N-P TAMI,relative to a scene detection step of a flowchart of FIG. 32.

FIG. 35 shows a detailed flowchart of coding steps relative to one ormore generalized coding steps shown in the flowcharts of FIGS. 32, 36A,and 36B.

FIGS. 36A and 36B each show portions of the processing steps associatedwith the "MAIN" step of the flowchart of FIG. 32, in another embodimentof the invention.

FIG. 37 is a flowchart showing details for the MEP step of the flowchartof FIG. 36A.

FIG. 38 is a flowchart showing the MEI step of the flowchart of FIG.36A.

FIG. 39 is a block schematic diagram showing a hardware configurationfor carrying out various embodiments of the invention.

FIG. 40 is a block schematic diagram showing a portion of a scene changedetector generally shown in FIG. 39.

FIG. 41 shows a block schematic diagram of a distance computation unitgenerally shown in the diagram of FIG. 40.

FIG. 42 is block schematic diagram of a Type 1 scene change detectorshowing generally in FIG. 40.

FIG. 43 is a block schematic diagram showing a Type 0 scene changedetector shown generally in FIG. 40.

FIG. 44 shows a block schematic diagram of a GOP structure generationunit shown generally in FIG. 40.

FIG. 45 shows a block schematic diagram of a scene detector controllermodule shown generally in FIG. 40.

FIG. 46 shows a block schematic diagram of a motion compensator moduleshown generally in FIG. 39.

FIG. 47 shows a truth table for a switch control block or module of themotion compensation shown in FIG. 46.

FIG. 48 shows a block schematic diagram of a motion estimator module forthe motion compensator shown in FIG. 46.

FIG. 49 shows a block schematic diagram of a telescopic motion estimatorcontroller shown in the schematic of FIG. 48.

FIG. 50 shows a block schematic diagram and a table for the bit ratecontroller module of the system shown in FIG. 39.

FIG. 51 shows a block schematic diagram for a scene change detectorconfiguration associated with the encoder of FIG. 39, for a BS E-TAMIembodiment of the invention.

FIG. 52 shows a block schematic diagram of a binary search unitassociated with the scene change detector of FIG. 51.

FIG. 53 shows a block schematic diagram of a generalized subband videoencoder incorporating TAMI for another embodiment of the invention.

FIG. 54 is a simplified diagram showing a multi-resolution motionestimation method associated with the encoder of FIG. 53.

FIG. 55 shows a block scan mode for a differential pulse code modulation(DPCM) scheme using horizontal prediction from the left, with verticalprediction for the first column.

FIG. 56 shows a block scan horizontal scan mode relative to the subbandencoding system of FIG. 53.

FIG. 57 shows a vertical block scan mode relative to the subbandencoding system of FIG. 53.

FIG. 58 shows a table of performance comparisons between variousembodiments of the invention.

FIG. 59 shows a block schematic diagram supplementary to FIG. 53 forproviding greater details of a system for the subband video encodingembodiment of the invention.

FIG. 60 shows a block schematic diagram of the subband analysis moduleof the subband video encoding system of FIG. 59.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION

One of the perceptual factors exploited in one embodiment of the presentcoding scheme is temporal masking, which is closely related to butdifferent from temporal summation in low level perception. There hasbeen little prior work in exploiting temporal masking for image coding.

Temporal summation has been known for over a century as Bloch's law. Thelaw says that below a critical time period or duration (T), about 100ms(milliseconds), luminance perception by the human eye is constant aslong as the product of time duration (T) and intensity (I) is keptconstant, namely:

    I×T=k                                                (1)

This describes a kind of temporal summation (integration) occurring inthe human visual system. FIG. 1 illustrates the temporal summationeffect, in which luminance perception depends only on the total area ofthe pulses 1, 2, or 3 regardless of the durations and the intensities ofindividual pulses. It is known that temporal summation is a neuralphenomenon and that it does not occur at the photoreceptor level, butresearchers still do not know whether it occurs at the ganglion celllevel, the lateral geniculate nucleus (LGN), or the cortex. The criticalduration is about 100ms when a spot of light is seen against a blackbackground (scotopic conditions). Under natural viewing conditions(photopic conditions), the critical duration is much shorter, on theorder of 20ms. For a task involving perception of image content, such asbeing able to tell object orientation in a test image, the criticalduration can be as long as several hundred milliseconds. Because of thissummation process, human vision has limited temporal resolution, and thecritical duration is generally not less than 20 to 100ms. This is themain psychophysical factor that influenced the choices of frame ratesfor movies and television.

The present invention is concerned with the temporal masking aspect ofvision. A simple low pass filter model is used to characterize thephenomenon. It is adequate to model human temporal processing as a leakyintegrator, i.e., a first-order low pass filter.

The temporal transfer function expressed as a Laplace transform, can bemodeled by: ##EQU2## where T is recovery time (critical duration), whichis about 100 to 200ms. The frequency response, expressed in terms of aFourier transform, is given by: ##EQU3##

This response roughly reflects the temporal modulation transfer function(MTF), which is defined as the reciprocal of the just visible sine wavemodulation amplitude. It is a sensitivity response function of the eyewith respect to temporal frequency of the stimulus. FIG. 2A shows afrequency response curve 27 of the leaky integrator model, and FIG. 2Bthe frequency response curves of a typical temporal MTF, with curves 31and 33 having spatial frequencies of 0.5 cycles/degree and 16cycles/degree, respectively.

There are two kinds of temporal masking, forward and backward. It isforward masking when arriving stimulus acts forward in time to effectone which comes later, and backward masking when the stimulus arrivinglater effects one which has already come and gone. Because of theseeffects in the coding scheme the immediate past frame at a scene changecan be coarsely coded, as can the following frame. This effect wasverified in an experiment, in which any difference of perceptual qualitybetween the original frames and the frames with a coarsely codedimmediate past frame was detected. Little perceptual difference wasdetected even when the frame is coded with as few as 20% of the numberof bits of a regular frame.

In general, these forward and backward masking effects can be explainedby two underlying processes in temporal masking. One is masking bylight, which occurs when there is a drastic change in luminance. Thiseffect is believed to be caused by lateral inhibition in the retina. Theother is masking by visual processing noise, which occurs when there isan abrupt change in visual patterns. This can be explained by thedegradation of spatial contrast by temporal summation. The combinedeffect of these two masking processes produces the forward and backwardmasking effects when there is a scene change.

Temporally Adaptive Motion Interpolation(TAMI) Algorithm Fixed Bit RateCoding(FBR-TAMI):

The present new motion interpolation coding techniques adopt someterminology from the MPEG standard, including I (intra frame), P(predicted frame), and B (bidirectional interpolated frame), and aregenerally compatible with apparatus following present video codingstandards. In one embodiment of the invention, a temporally adaptivemotion interpolation algorithm (hereinafter referred to as TAMIalgorithm) was developed. One variation of this algorithm uses fixed bitrate coding, providing a FBR-TAMI, which is discussed below.

In the TAMI algorithm, the interval between two reference frames isadapted according to the temporal variation of the input video, and thebit rate control takes advantage of temporal masking in human vision.The crucial issue in this approach is the bit rate control problembecause the group of pictures (GOP) structure is dynamically changing.When a scene change occurs, it is desirable to code the first frameafter the scene change as an I or intra frame, which might beimpractical because the bit rate would increase drastically if therewere many such scene changes in one GOP.

This problem can be resolved by coarsely quantizing the new I or intraframe with the same number of bits as used for a regular B frame. Thisdoes not degrade the picture quality when the sequence is continuouslydisplayed because of the forward temporal masking effect. It is knownthat if the bit rate (bandwidth) of frames following a scene change isgradually increased back to full bit rate (bandwidth), then thedegradation of the frames following a scene change is not perceptible.

Using a poor quality intra frame after a scene change directly affectsthe picture quality of the following frames until a new intra frame isused, with the quality of the following frames becoming better oversuccessive frames. This gradual improvement in quality is therebyachieved without a complex scheme for explicitly controlling bitallocation on a frame-by-frame basis.

To detect significant temporal variations of the input video, differenttemporal distance measures were considered in developing the presentinvention. These distances are actually a measure of the amount ofglobal motion between frames. These motion measures can be determined bythe difference of histogram (DOH), histogram of difference (HOD), blockhistogram (BH), block variance (BV), and motion compensation error(MCE), respectively. They are described in detail below. In the presentTAMI algorithm, six different frame Types, I1, I2, P1, P2, B1, and B2are used. Frame Types I1, P1, and B1 are the same regular frame types asdefined in the MPEG standard. Frame Types I2 and P2 have the same bitallocation as B1 frames; thus I2 and P2 are very coarsely quantizedintra and predicted frames, respectively. On the other hand B2 is aninterpolation frame with more bits than a regular B1 frame, andgenerally fewer bits than a P1 frame. An I1 designated frame is a fullframe, and is finely quantized.

In one embodiment for TAMI, an I1 frame is always the first framedesignation for each GOP. When the cumulative motion or measureddistance from an immediately preceding reference frame and a successiveframe in a GOP exceeds a Type 0 threshold, the immediately prior frameto the successive frame is designated as a P1 frame. When the motion ormeasured distance between two successive frames exceeds a Type 1threshold, the first or immediately prior frame is designated as a P2frame, and the second or immediately past frame as an I2 frame. I1, I2,P1, and P2 frames are reference frames used in interpolating B1 and B2frames. In a GOP where Type 0 scene changes occur, B1 frames aredesignated between reference frames. In a GOP where no Type 0 scenechanges occur, B2 frames are designated between reference frames, forexample, as described below in greater detail. In other words, B2 framesare used when no Type 0 scene changes are detected in a GOP, whereby noP1 frames other than possible default P1 frames result. Accordingly, thebits saved by not requiring additional P1 frames may be distributedamong B1 frames to cause them to become B2 frames with greaterresolution. Accordingly, the B2 frames are basically B1 frames withslightly higher bit allocation.

In another embodiment of the invention, in a GOP a first occurring Pframe is predicted from the immediately previous I frame. A subsequentlyoccurring P frame is predicted from the immediately previous P frame orI2 frame, whichever is closest.

FIG. 3 shows the block diagram 10 for the TAMI algorithm. The TAMIalgorithm first looks at all the frames in the current GOP, detectsscene changes of Type 1 (using a Type 1 scene change detector or SCD12), and detects scene changes of Type 0 (using a Type 0 scene changedetector or SCD 14). The next steps determine the positions of P and Bframe in the GOP or group of pictures structure (using a GS outputdetector 16). Then using the positions of P and B frames, the frames areprocessed by a motion compensated interpolation encoder 18, which issimilar to a typical motion compensation encoder except that it usestelescopic motion vector search and a different bit rate controlmechanism, in this example. As shown, encoder 18 includes a discretecosine transform (DCT) module 4, a quantizer (Q) 5, a variable lengthcode generator 6, a buffer 7, an inverse quantizer (Q⁻¹) 8, an inversediscrete cosine transform module (IDCT) 9, a motion estimator (ME) 15,and first and second summing junctions 17 and 19. Note that DCT 4converts data from the time domain into the frequency domain, as isknown in the art.

The flowchart 10 of the TAMI algorithm is illustrated in FIG. 4,including details of the programming for scene change detectors 12 and14. The algorithm 10 includes steps 100 through 110, and 16, forprogramming a microprocessor, for example, to take the followinggeneralized steps, using one GOP as a processing unit:

A. It loads frames from one GOP into an associated frame memory includedin the microprocessor (steps 101 and 102).

B. It detects the positions of scene change of Type 1 via the Type 1 SCD12 (steps 103, 106, and 107).

C. It detects the positions of scene change of Type 0 via the Type 0 SCD14 (steps 104, 108, and 109).

D. It determines the GOP structure, i.e., the positions of all I, P, Bframes via GS 16 (steps 105 and 16).

E. It outputs the information generated in step "D" to the motioncompensating encoder 18.

Two types of scene detectors 12 and 14 are required for processing thealgorithm, as shown. In FIG. 4, the first detector 12 declares a scenechange of Type 1 for the current frame when the distance or relativemovement measure between the current frame f_(c) and the immediate pastframe f_(c-1) is above a threshold T₁ (step 103). This type of scenechange corresponds to an actual scene content change; it is coded as anI2 frame (very coarsely quantized intra frame), and the immediate pastframe f_(c-1) is coded in step 106 as a P2 frame (very coarselyquantized predicted frame). The I2 frame coding exploits the forwardtemporal masking effect, and the P2 frame coding takes advantage of thebackward temporal masking effect.

The second detector 14 detects scene changes of Type 0. This implementsa temporal segmentation algorithm for processing. This algorithm, asshown in FIG. 4, declares the current frame f_(c) as a scene change ofType 0, when the distance or relative movement measured between thecurrent frame f_(c) and the last reference frame f_(ref) is above athreshold T₀ (see step 104). This time the immediate past frame f_(c-1)becomes a P1 frame which is a regular predicted frame. The bitallocation strategy for the temporal segmentation is that every endframe of temporal segments should become a P1 frame, and that the framesin between should be B1 or B2 frames depending on whether the extra P1frame is being used or not.

As a result of experimentation, one modification was made. The number ofextra P1 frames was set to one because more than two extra P1 frames wasfound to cause quality degradation, while zero extra P1 frames means noadaptability. The reason for the degradation when using many P1 framesis explained as follows. Since fixed bit rate coding with a feedbackloop 21 between the Buffer 7 and the quantizer (Q) 5 is being used (seeFIG. 3), the bit rate for successive P1 frames is gradually reduced(i.e., coarsely quantized) by the feedback loop 21. However, use of thecoarse P1 frame produces adverse effects. It degrades not only thequality of the P1 frames but also the B frames in between. It ispreferred to limit the number of coarsely quantized P1 frames inhigh-motion segments, whereby the P1 frames eliminated are replaced byB1 frames. The degradation effect with many or unlimited P1 frames canbe easily seen in FIG. 5, in comparison with the limited P1 scheme,relative to curves 23 and 25, respectively.

Two different bit allocation schemes may be used. The first scheme is aconstant bit allocation scheme where the bit allocation for each picturetype except B is always constant from GOP to GOP regardless of thenumber of detected SSPs (scene segmentation points, i.e. scene changesof Type 0). The constant bit allocation scheme is more suitable for someapplications for two reasons. First, the picture quality of I and Pframes varies more than the quality of a B frame does when a variablebit allocation scheme is used. Second, since an I frame is repeatedevery 15 frames (every 1/2 sec), a problem of 2 Hz blinking may beencountered if the I frame bit allocation is allowed to vary. The secondscheme is a variable bit allocation scheme where the bit allocation foreach picture type is allowed to change from GOP to GOP according to thenumber of detected SSPs, but the ratio of bit allocations betweendifferent picture types is always constant. When the constant bitallocation scheme is used, one cannot afford high variability of thenumber of P frames because of the constrained bit budget (fixed bit ratecoding or constant bit rate coding). Hence, to have some adaptability,in one embodiment the variation of the number of P frames is limited to1, which can easily be implemented by using two different bitallocations for B frames (B1 and B2 frames), i.e., B1 frames are used ifan extra P frame is used and B2 frames are used if not. The variable bitallocation scheme is used in the adaptation of the number of P frames,as described later, to allow a variable number of P frames in a GOP. Thefollowing paragraphs describe constant bit rate allocations in greaterdetail.

Fixed Bit Rate Coding (FBR TAMI):

The present TAMI algorithm uses a simple rate control strategy which isbased upon regulation of the number of bits inputted to a buffer 7 (seeFIG. 3), whereby if during coding of one frame the buffer 7 begins tofill up, the system is programmed to automatically reduce the number ofbits generated in coding the next frame. After the bits are allocatedvia GS output 16 for the different picture Types (I1, I2, P1, P2, B1,B2), the target bits for a slice are computed via encoder 18. Note thata slice is defined in the MPEG standard as a series of macroblocks (oneof the layers of the coding syntax). Also, a macroblock is defined inthe MPEG as "the four 8-by-8 blocks of luminance data and the twocorresponding 8-by-8 blocks of chrominance data coming from a 16-by-16section of the luminance component of the picture." At the end of eachslice, the buffer 7 content is monitored. If it is above or below thetarget buffer content, the QP (Quantization Parameter) is reduced orincreased respectively (slice level bit rate control). To be adaptive tothe changing coding difficulties of an incoming video sequence, the bitsproduced in the current picture are used as the target bits for the nextoccurrence of a frame of the same picture type. When the bits producedfor the previous frame are more or less than the target bits, the bitallocation for the next picture is adjusted to maintain an acceptablerange of bit rate per second by the equation (frame level rate control):##EQU4## where TB is target bit allocation, ABR_(GOP) is actual GOP bitrate, and TBR_(GOP) is target GOP bit rate, and XTB is the target bitallocation for the previous frame.

There is a difficulty with the motion estimation because the frameinterval between the current and the reference frame can be as large as15 (for GOP size 15), which means that the search region for a motionvector can be as large as 105, assuming the search region for adjacentframes is 7. Using Mean Absolute Difference (MAD) as the matchingcriterion for the full search, this would require about 2.5×10¹¹operations per second for a sequence in the CIF format (352 by 240pixels, 30 frames/sec). To reduce this computational complexity, atelescopic search as mentioned in the MPEG standard is used because itis believed to be the most suitable for the long interval betweenreference frames, and provides very good motion compensating performancein computer simulations. An example to search motion vectors betweenfour frames is given in FIG. 6. The basic idea of the telescopic searchis that the center position of the search window for a block in thecurrent frame is shifted by the motion vector obtained for acorresponding macroblock in the previous frame, repeating until the lastframe is reached. Accordingly, as shown, vector 22 shifts the center ofthe search window from the center of block 20 to the center of block 24in Frame 1; vector 26 shifts the center of the search window from thecenter of block 24 to the center of block 28 in Frame 2; and vector 30shifts the center of the search window from the center of block 28 tothe center of block 32 in Frame 3.

N-P TAMI algorithm:

There are two problems in the TAMI algorithm. One is its tendency toproduce longer intervals between reference frames. This promotesperceptually displeasing coding errors around moving edges, which iswell known to be typically associated with any block motion compensationcoding. This error occurs because a macroblock may consist of twoobjects (usually, one is a moving object and the other is background)which are moving in two different directions. Since each block is motioncompensated by a single motion vector, this produces residuals havingmainly high frequency components that are lost in the quantization stagebecause of coarser quantization for high frequency DCT coefficients. Theother problem in the algorithm is that longer delay is inevitablebecause all the frames in a GOP have to be stored before the TAMIalgorithm runs. The longest encoding delay in this case is one GOP size,which is usually fifteen frames, correspondingly to 1/2 second when theframe rate is 30 frames/sec.

To alleviate these problems, generalizations of the TAMI algorithm orprogramming steps were developed. To reduce the encoding delay anddistances between reference frames, N P1 frames are inserted into GOPstructure by default. The modified program is herein referred to as theN-P TAMI programming steps or algorithm. Note "N" is the number ofdefault P frames. The modified program allows for the choice of N from 0to 3, and produces four different schemes, namely, 0-P scheme(IBBBBBBBBBBBBBB), 1-P scheme (IBBBBBBBPBBBBBBB), 2-P scheme(IBBBBPBBBBPBBBB), and 3-P scheme (IBBBPBBBPBBBPBBB). For even N, theGOP size must be fifteen frames to have even spacing between referenceframes. For odd N, the GOP size should be sixteen frames for the samereason. FIGS. 7A through 7D show the default GOP structures for N=0, 1,2, and 3, respectively. Note that for N=4, this is equivalent to theconventional implementation of the MPEG standard. FIG. 8A also shows anexample of the GOP structure generated by the present 1-P TAMIalgorithm, when there is a scene change of Type 1, and at least onescene change of Type 0. Note that as described above, B1 frames aredesignated between any pair of reference frames, i.e., I₁, I2, P1,and/or P2, in this example. If no Type 0 scene changes are detected, asshown in FIG. 8B, for example, B2 frames are designated betweenreference frames, in that at least one less P1 frame is designatedrelative to the example of FIG. 8A, permitting more bits to be used forthe B frames. As N increases, smaller encoding delay and smallerinter-reference frame intervals are encountered, but bit rate savingsare reduced.

Assume that bit allocations (Kbit/frame) are |I1|=180.0, |I2|=6.75,|P1|=100.5, |P2|=6.75, |B1|=6.75. The relationship between B2 and B1 isas follows: ##EQU5## where N is the number of P1 frames used, and M isthe GOP size.

The bit rates per second, BR, for the four schemes are derived, forexample, from the allocations via the following computations: ##STR1##Variable Bit Rate Coding(VBR-TAMI):

Spurred by the recent advancement of the ATM (Asynchronous TransferMode) concept of B-ISDN (Broadband Channel Integrated Services DigitalNetwork) technology, variable bit rate coding (packet video) is becominga very promising scheme for network-oriented video transmission. Thisscheme relaxes the bit rate control restrictions on the encoder andenables constant picture quality transmission instead of constant bitrate.

In the above-described fixed bit rate coding embodiment for FBR-TAMI,the number of P1 frames is limited because of the fixed output bit rateconstraint. As a result, the output bit rate is maintained at the costof degradation of picture quality in intervals where there is highmotion activity. If the restrictions on the number of P1 frames and thefeedback loop for bit rate control are removed, the TAMI algorithmbecomes a VBR (Variable Bit Rate) encoder and produces constant picturequality (perceptual picture quality, not in terms of constant SNR) byinserting more P1 frames into temporally busy regions. Hence, theVBR-TAMI encoder will compress video data much more than a conventionalfixed GOP structure encoder for the FBR-TAMI encoder.

FIG. 9 shows the block diagram of one embodiment of the present VBR-TAMIencoder 34. Compared to the FBR-TAMI encoder 18 of FIG. 3, the VBR-TAMIencoder 34 does not include a buffer 7, or a rate control feedback 21for maintaining a fixed bit rate. Instead, the network 36 acts as abuffer, and a network estimator 35 is included in a network feedbackloop 37 between network 36, and quantizer 5 and variable length coder 6.

The flowchart for the VBR-TAMI algorithm 38 is shown in FIG. 10. Thenump1 statement in step 104 of the FBR-TAMI algorithm 10 of FIG. 4, forlimiting the number of P1 frames, is not required in step 104' of theVBR-TAMI algorithm 38 of FIG. 10. This is the only difference betweenalgorithms 10 and 38. The use of B2 frames in the VBR-TAMI algorithm 38is meaningless because use of any number of P1 frames is now allowed.

In the VBR-TAMI algorithm 38, when there is a scene change of Type 1,temporal masking is also applied to the two frames at the scene change(i.e., the preceding frame is a P2 frame and the following one is an I2frame).

The following is a simple bit rate performance analysis for the VBR-TAMIencoder 34. For simplicity assume that the P1 event (i.e., declarationof scene change Type 0) is a Bernoulli random variable. Then the numberof P1 frames in a GOP is a random variable K with a binomialdistribution, as shown by the following equation:. ##EQU6## where M isthe GOP size, k is the number of P1 events, and p is the probability ofhaving a P1 event at a frame. The mean of this distribution isk=E[K]=(M-1)p. (M-1) is the number of possible positions for P1 becausethe first frame of a GOP always has to be an I1 frame. Although it isnot exactly correct because of the exclusion of the first frame in aGOP, the interarrival distribution of P1 arrivals can be modeled by ageometric distribution as follows:

    P(T)=p(1-p).sup.(T-1)                                      (7)

where T=0, 1, 2, . . . is the interarrival time between successive P1frames. The mean of this distribution is given by E[T]=1/p, and p=k(M-1)from the binomial distribution. From this one obtains k=(M-1) E[T].

The mean and variance of the output bit rate with k as a parameter cannow easily be computed, to obtain a rough measure for motion activity inthe input video. For example, assume there are kP1-frame events in aGOP. Then one will have one I1 frame, kP1 frames, and (M-1-k) B1 frames,and the bit rates are computed as follows: ##EQU7## where R_(GOP)(k) isthe GOP bit rate, R_(i1), R_(p1), and R_(b1) are bit allocations for I1,P1, and B1 frames, and R(k) is the bit rate per second. Then the meanand variance of the bit rate are given by the following equations:##EQU8## where R₀ =R_(i1) +(M-1) R_(b1), which is the bit rate whenthere is no P1 frame. This shows that the mean bit rate is linearlyproportional to the expected number of arrivals, and that the varianceis at its maximum when the arrivals are the most uncertain (i.e.,p≈1/2).

As an example, when the M=15, k=2, R_(i1) =180, R_(p1) =100.5, andR_(b1) =6.75, provides E[R]=924Kbit/sec and σ² [R]=245.5Kbit/sec.Similarly for the 1-P scheme, the following equations apply: ##EQU9##

The following expressions for mean E[R], and variance σ², also apply:##EQU10## where R_(o) (1)=R_(i1) +lR_(p1) +(M -l-1) R_(b1), which is thebit rate when there are l P1 frames.

Distance Measures for Temporal Segmentation:

Five different distance measures for temporal segmentation will now beconsidered, as an example. First, notation must be defined. The numberof pixels of an image is denoted by N_(pix) ; the width by W; the heightby H; the number of luminance levels by q; and the frame number index byn. Then an image sequence is defined by

    F={f.sub.n |f.sub.n : L.sub.x ×L.sub.y →F,n=0, 1, 2, . . .}                                                    (16)

where L_(x) ={0, 1, . . . , W-1}, L_(y) ={0, 1, . . . , H-1}, and F={0,1, . . . , (q-1)}. The corresponding histogram sequence is defined by:

    H={h.sub.n |h.sub.n : F→Z.sup.+,n=0, 1, . . . ,(q-1)}(17)

where Z⁺ is a set of all nonnegative integers. The histogram operator Hfrom an image to a histogram is defined as:

    h.sub.n =Hf.sub.n                                          (18)

where H :F→H.

1) Difference of histograms (DOH): The distance measure between f_(n)and f_(m) is defined by I1 norm of their histogram difference asfollows: ##EQU11##

Researchers have reported that the luminance histogram is a veryefficient index for image content. The histogram difference between twopictures can thus be a good measure of the correlation of the contentbetween them. Another important advantage of using DOH distance measureis its insensitivity to local motion activities, regardless of the speedof the motion (e.g., an object moving in a stationary background),compared to its sensitivity to global motion activities such as zooming,panning and scene changes, since a good temporal segmentation shouldeffectively detect global changes and not be too sensitive to localmotion activity that can be compensated for by a typical motionestimation algorithm.

The DOH is better for detecting global changes rather than for detectinglocal motion.

2) Histogram of difference image (HOD): The histogram of differencesbetween two images is denoted by:

    HOD(·)=H(f.sub.n -f.sub.m)                        (20)

where HOD is a function defined as hod:

{-(q-1),-(q-2) . . . ,-1, 0, 1, . . . , . . . ,q-1}→Z⁺.

Note that this is essentially the same quantity as the summation of theentries of the co-occurrence matrix along lines parallel to thediagonal. If there are more pixels far from the origin of HOD, it meansthat there are more changes in the image. The movement criterion can beroughly defined by the ratio of the counts at nonzero positions to thetotal number of counts in HOD. Hence the distance measure is defined asfollows: ##EQU12## where α is a threshold for determining the closenessof the position to zero. This HOD measure has somewhat differentcharacteristics than DOH. HOD is much more sensitive to local motionthan DOH.

3) Block histogram difference (BH): In HOD, a problem is that localmotion information is lost by computing a global histogram. This problemcan be reduced by computing histograms of each block, and summing theabsolute difference of each block histogram between 2 frames. Let thetotal number of macroblocks per frame be denoted by mbnum. For a givenb_(th) macroblock of frame f_(n), the block histogram is defined asfollows:

    h.sub.n (b,·)=H.sub.b f.sub.n                     (22)

where H_(b) is the histogram generator for the b_(th) macroblock, andbε[0, 1, . . .,mbnum-1]. The distance measure is defined by: ##EQU13##where bε[0, 1, . . . ,(mbnum-1)] is the index number for a macroblockand iε[0, 1, . . . ,(q-1)].

4) Block variance difference (BV): The idea of using this measure is thesame as for the block histogram difference except that the variance isused instead of the histogram. The distance using the sum is defined bythe sum of absolute difference of the block variance between two frames,which is given by: ##EQU14## where bε[0, 1, . . . ,(mbnum-1)]. Like theblock histogram difference, this approach is made sensitive to localmotion activities by computing the differences block by block

5) Motion compensation error (MCE): Suppose frame f_(m) is predictedfrom f_(n) by motion estimation. Since coding difficulty is directlydetermined by the error between f_(m) and f_(m) which is a predictionfrom f_(n). This motion compensation error can provide a measure for thecoding of the error image between f_(m) and f_(m). Hence the distancemeasure using this error is defined by the following equation: ##EQU15##

Since this measure is computed directly from the prediction error, it isthe nearly ideal measure for the coding difficulty of prediction error.However, the best measure would be the number of bits generated by thisimage coder, but this is a unrealizable because it would require theencoding results in the preprocessing stage. This approach using motioncompensation error is near-optimal but the drawback is that it iscomputationally too expensive.

Optimal Spacing Algorithm:

In another embodiment of the invention, the basic TAMI algorithm wasimproved for the detection of scene change Type 0. It was recognizedthat since a fixed GOP size is being used, the basic TAMI algorithm maynot produce the best possible spacing between reference frames. Adescription follows for the modification developed to improve thespacing, by providing an optimal spacing algorithm, in a preferredembodiment. As will shown, in using the optimal spacing algorithm (OSA),Type 0 scene change detectors are not used; only Type 1 scene changesare detected.

The flowchart of the OSA algorithm 60 for a 2-P scheme is given in FIG.11. The algorithm 60 takes the following steps:

1. In step 61, GOP frames are loaded into memory, and also differencemeasures for each frame in a GOP are generated within step 61.

2. Using a distance measure between two adjacent frames, scene changesof Type 1 are detected in step 62. The frame just before the scenechange is declared as a P2 frame, and the frame just after the scenechange as an I2 frame (i.e, temporal masking is also being used in thisscheme). Note that FIGS. 4 and 10 show the steps 106 and 107 foraccomplishing the determination of P2 and I2 frames via a Type 1 scenechange detector. The P frames corresponding to these points are notincluded in the total number of P frames (i.e. P1 frames).

3. An exhaustive search (steps 63-69) is used to find the best positionsof P1 frames for equidistance, i.e., minimizing the deviation from themean of the distances between the positions that include candidate P1frames that would have been designated by Type 0 detection and thepoints of scene change Type 1, from which the GOP structure isdetermined in step 70.

The deviation from the mean in step "3" above can be defined using thefollowing notations. Suppose that the GOP is partitioned into s segmentswhere each segment consists of two reference frames. Define the firstand the last frame numbers of the ith segment by fpn(i) and lpn(i). Thedistance for the ith segment can be expressed as:

    d.sub.i =D(f.sub.fpn(i), f.sub.lpn(i))                     (26)

where D is the distance measure. Then the deviation, dev is: ##EQU16##

FIG. 12 shows an example of the GOP structure using 2-P optimal spacingalgorithm when there is a scene change of Type 1.

Assume that the GOP size is equal to M; the number of P1 frames used isN; the number of scene changes of Type 1 is u, and the number ofadjacent pairs of scene changes of Type 1 is v. Then the number ofsearches S is as follows: ##EQU17## since it searches through allcombinations of N frames of P1 type. The number of positions searched is(M-2u+v-1) because the first frame is always coded as an I frame and thetwo neighboring frames for each scene change of Type 1 are excluded.However, if there are v pairs of scene changes there are v commonneighboring frames, so the total number of exclusions due to scenechanges becomes (2u-v). Since N is a fixed number, less than five, thisis a polynomial-time search algorithm. The GOP size M is usually lessthan 20 and (2u-v) is always positive, whereby in such instances even anexhaustive search is not computationally too expensive (S is roughly onthe order of 100). If there are several optimal answers, a solution ischosen where the intervals (frame number difference) between referenceframes are the most evenly distributed, i.e., a solution where thedeviation from the mean of interval sizes is the smallest.

The OSA algorithm (Optimal Spacing Algorithm) can in another embodimentalso be improved further in its adaptivity by using one extra reservedP1 frame as in the TAMI algorithm. Here B2 frames are used depending onthe local motion activity. If the average distance between referenceframes is above a threshold, one more P1 frame is assigned by default(i.e., it becomes an [(N+1)-P] scheme using B1 frames). Otherwise the B2frames are used (an N-P scheme using B2 frames). Further improvement maybe obtained by adapting the number of P1 frames to a criterion of localmotion activity.

In steps 61-66 (FIG. 11) of this OSA algorithm 60 the computation is notoverly expensive when the distance measure uses histogram or varianceapproaches, i.e., the order of the number of operations per frame isO(N_(pix))≈10⁵, where N_(pix) is the number of pixels, resulting inabout S×10⁶ ≈10⁸ operations for an OSA algorithm with 352×240 imagesize, where S is the number of searches given in equation 29. However,if a motion compensation error approach is used (see Equation 25 infra),the complexity becomes about S×10¹¹ ≈10¹³, assuming S is 10². Hence afast motion vector search algorithm has to be used to make the OSAalgorithm practical.

Highly accurate motion vector information for segmentation is notrequired, therefore the computation can be reduced by a factor of 2 inboth dimensions via pixel subsampling , i.e., about 1/4 computationsaving is achieved. Extra savings can be obtained by using a backwardtelescopic search, as shown in FIG. 13, where the sequence of the searchruns backward in time opposite to the direction of conventionaltelescopic search (see FIG. 6). Note that the accuracy of the backwardmotion vector is better than the usual forward telescopic search. Theprevious motion vector used as a prediction for a current search is morecorrelated to the current macroblock in the backward search than in theforward search, because the current macroblock to be matched is alwaysfixed, whereas that of the forward search is always changing in theforward search.

Experimental Results:

FBR-TAMI algorithm:

For testing of the FBR-TAMI algorithm 10 (see FIG. 4), simulations wereperformed using a tennis sequence in the CIF format (352 by 240 pixels)for different N-P schemes for N=0, 1, 2, and 3. The Huffman codingtables in the MPEG standard MPEG91 were used for the variable lengthcoding. The difference of histogram(DOH) was used for the distancemeasure in the simulations for the TAMI algorithm because of itssimplicity.

FIG. 5 shows the SNR's for the unlimited P1 (VBR TAMI) and limited P1(fixed TAMI) embodiments. It shows that using a B1 frame is better thanusing a P2 frame, even when many scene changes of Type 0 are detecteddue to very busy motion activity. Hence for the rest of the simulationsexcept for VBR-TAMI results, the limited P1 scheme is used. FIGS.14(a)-14(e) show SNR results for the temporally smooth region, and itscorresponding bit rate per frame is also provided in FIGS. 15(a)-15(e).More specifically, FIGS. 14(a)-14(e) show respective SNR vs. frames orimages with little motion, for a tennis scene at an average bit rate of736.5 Kbit/sec for conventional 4-P, and 0-P, 1-P, 2-P, and 3-P schemes,respectively. FIGS. 15(a)-15(e) show the bit rates for images withlittle motion corresponding to FIGS. 14(a)-14(e), respectively. FromFIGS. 14(a) and 14(e), from amongst the several respective N-P schemes,the 1-P scheme performs the best in term of SNR and subjective quality,and its SNR is about 1dB better than the conventional 4-P fixed scheme.FIGS. 16(a) though 16(e) show the results for the high temporal activityregion with an abrupt scene change, and the corresponding bit rateresults are given in FIGS. 17(a) through 17(e), respectively. The SNR ofthe 1-P scheme is also slightly better (by 0.5dB) than that of theconventional 4-P scheme even in the case with a scene change. Thedecoded image quality of the 1-P scheme has been determined to comparefavorably with that of the conventional method 4-P scheme. Theperformance improvement using the 1-P scheme is more noticeable for lowbit rate coding (300 Kbit/sec - Tennis), and the SNR and the bit rateresults for the target bit rate are given in FIGS. 18(a)-18(e) and FIGS.19(a)-19(e), respectively. Averages of SNR's and bit rates for FIG. 14,FIG. 16, and FIG. 18 are given in tables shown in FIGS. 20, 21, and 22,respectively. They show that the present 1-P FBR-TAMI scheme outperformsthe conventional fixed 4-P scheme in most of the cases.

VBR-TAMI algorithm:

FIG. 23(a) and FIG. 23(b) show SNR and its bit rate comparison,respectively, between VBR-TAMI and 0-P FBR-TAMI, where they have thesame average bit rate of 663Kbit/sec, in this example. Hence there isless temporal variation in the picture quality. As expected, the SNR forVBR-TAMI is more stable than FBR-TAMI. At the cost of variable bit rateoutput, the present VBR coding scheme can handle scenes with rapidmotion without significant loss of quality.

Optimal spacing algorithm:

FIGS. 24(a)-24(e) show relative distances between a current frame andthe first frame (frame 120) using the five different measurementmethods, e.g. DOH, HOD, BH, BV, and MCE methods of measurement,respectively. In the plots, it can be seen that the DOH and MCE criteriaare more sensitive to global motion rather than to local motion, whilethe other three criteria are sensitive both to global and to localmotion as discussed above. More specifically, FIGS. 25(a)-25(e) showcurves for SNR vs. frame number for an OSA using DOH, HOD, BH, BV, andMCE methods of distance measurement, respectively. HOD performs the bestbecause its sensitivity to local motion is more important when there isno global motion between frames 120 and 180, only some local motion.

The table of FIG. 26 shows that the MCE criterion produces the bestoverall performance for various motion activities. In the table, frames89-149 represents a scene with motion, frames 120-180 a scene with anabrupt scene change at frame 148, and frames 0-60 a scene with very highglobal motion (zooming out from frame 30 to 60). The good performance ofMCE may be because MCE is the nearly ideal distance measure and isexpected to be more robust to various motion activities.

FIGS. 28(a)-28(e) shows SNR results using the adaptive optimal spacingalgorithm with B2 frames for the five different distance measures DOH,HOD, BH, BV, and MCE, respectively. BH performs the best because it isalso a good measure of local motion and there is no global motionbetween frames 120 and 180. However, the table in FIG. 27 shows that theBH criterion produces the best overall performance for various motionactivities. Comparisons between the tables of FIGS. 26 and 27, showsthat the performances with different distance measures are similar withone another. Comparisons between the tables of FIGS. 20 and 27, showsthat the performances of FBR-TAMI and OSA are also similar, with slightdifferences depending on what kind of distance measure is used.

As illustrated and described above, the present invention provides thatpositions of video reference frames are made adaptive to input videomotion activities, and bit rate control is used to exploit maskingeffects in visual perception. Three major different algorithms,FBR-TAMI, VBR, and OSA, are presented, and shown to compare favorablyover conventional motion interpolation coding with fixed GOP structures.Although FBR-TAMI and OSA are similar in their performances, TAMI haslower algorithmic complexity. The trade-off in this approach is that ascheme with a lower number of predicted frames has a better compressionratio at the cost of larger encoding delay. Embodiments of the presentinvention are expected to be useful in several different areas such asthe variable bit rate coding, low bit rate coding, coding for storage onCD-ROM, and temporally adaptive 3-D sub-band coding, for example. TheFBR-TAMI algorithm is suitable particularly for low bit rate coding suchvideo conferencing or video phone where the rarity of rapid motion isvery advantageous, and it is also suitable for storage application likeCD-ROM where relatively large delay in encoding can be tolerated.

In FIG. 29, a composite of three curves shows a comparison between theTAMI and OSA embodiments of the invention relative to image movement.The uppermost curve 120 shows a plot of image movement versus framenumber for a GOP of 15 frames. In this example, the image movement curve122 shows a region 124 of "busy temporal activity" between frames 1 and7, and a region 126 of "low temporal activity" 126 between frames 8 and15. As shown, in region 124 P frames occur more frequently or are closerto one another in this region because there is more data change, that isthere is greater image movement from one frame to another. Contrariwise,in region 126 where image movement is substantially less, the P framesoccur less frequently, or are further apart from one another, becausethere is less data change or image movement from one frame to another.In the curve section 128, TAMI processing for coding frames is shown asa plot of frame distance, that is the global picture movement betweenframes relative to frame number. The frame distance or movement at whicha Type 0 threshold is detected is shown by the broken line 130. Asshown, each time the frame distance or image movement between framesexceeds the Type 0 threshold 130, the immediately previous frame fromthe occurrence of the Type 0 threshold is designated as a P2 frame. Aspreviously explained, in this example, a GOP consists of 15 frames,designated by frame numbers "0" through "14", with the "15th" designatedframe actually being the first frame of the next GOP. The first frame isalways designated as an "I" frame. The next frames located between anytwo reference frames, such as P frames and I frames are designated "B"frames, as shown. Note that when using the TAMI processing as shown incurve section 128, the P frames are further apart in the region of lowtemporal activity 126, relative to the region of busy temporal activity124. By using OSA processing, as shown in the curve section 132, certainof the P frames designations are changed to shift right or left formaking the P frames as equidistant from one another as possible.Accordingly, as shown, TAMI designated frame 10 as a P frame, whereasthrough OSA processing, in this example, the P frame is shifted to theleft from frame 10 to frame 9. Similarly, in TAMI curve 128, frame 13 isdesignated as a P frame, whereas through OSA processing, the P frame isshifted from frame 13 to frame 12, as shown in curve section 132. Alsoas a result of this shifting, frame 9, designated as a B frame in TAMI,is designated as a P frame in OSA, frame 10 designated as a P frame inTAMI is designated as a B frame in OSA, and frame 13 designated as a Pframe in TAMI is designated as a B frame in OSA. As a result, the Pframes in region 126 are more equidistantly spaced from one another,making more efficient use of bit allocations via OSA, relative to TAMI.

Adaptive selection of the number (N) of reference frames:

A bit rate control algorithm, mainly for TAMI, to allow variable N foranother embodiment of the invention will now be described. Note that asindicated above, N is the number of P1 frames. One simple approach toadapt N is to use a constant threshold for detection of a Type 0 scenechange, and use one P frame for each detected. To adapt N subject to afixed bit rate constraint, a variable bit allocation as described aboveis used. The target bit allocations for different picture types areupdated according.

To describe the algorithm, let the channel bit rate (bits/sec) bedenoted by R, GOP size by M, expected GOP bit rate by G, and target bitallocation for picture Type t by D_(t). The bit allocations for I1, I2,P1, P2, and B1 frames: D_(I1) =C_(I1) x, D_(I2) =C_(I2) x, D_(P1)=C_(P1) x, D_(P2) =C_(P2) x, D_(B1) =C_(B1) x, respectively, where x isa common factor and C_(I1), C_(I2), C_(P1), C_(P2), and C_(B1) areconstants for I1, I2, P1, P2, and B1 frames, with C_(I2) =C_(P2)=C_(B1). Unlike the constant bit allocation scheme described above, xnow is allowed to vary from GOP to GOP, thereby providing the presentvariable bit allocation scheme. Note that B2 frames that were used for alimited variation of N by 1 are not required. The common factor x isdetermined by the relationship, R=2G where G=(C_(I1) +NC_(P1)+(M-N-1)C_(B1))x. The following formula for target bit allocationresults: ##EQU18##

This bit allocation is updated by use of equation (30) at the beginningof each GOP.

A fast heuristic approach for positioning (BS E-TAMI):

Another embodiment of the invention designated BSE-TAMI (Binary SearchEquidistant TAMI will now be described. Assume N SSPs (scenesegmentation points or Type 0 scene changes) are detected by the scenechange detection algorithm 14 (see FIG. 4) using a constant threshold.Assume that the distance measure is an integer and, as a basis fordeveloping a heuristic, is a monotonically increasing function withrespect to the time separation between two frames. HOD (histogram ofdifference) is used in such a simulation to measure motion by distancemeasurements, because it generally tends to be monotonic.

The problem is to find nearly equidistant positions of SSPs or Type 0scene changes. The present fast heuristic search is for positions thatare close to the best positions. FIG. 30 is an example where two SSPs orType 0 scene changes are detected by an SSP detector 14 using an initialthreshold, τ₀, which produces N SSPs. Denote the distance between thelast SSP and the end frame of a GOP by a (τ). Also denote a₀ =a(τ₀). Theproblem is to start with τ₀ >a(τ₀)=a₀ and to find the smallest τ in [a₀,τ₀ ] satisfying τ≧a(τ). Since the distance measure is assumed to bemonotonic, a(τ) either increases or stays constant as τ decreases fromτ₀. More specifically as τ decreases from τ₀, eventually τ≦a(τ) will beattained. In other words, point A crosses or reaches point B at somepoint as the threshold is decreased (see FIG. 30).

Since τ is an integer and has a finite range of [a₀, τ₀ ], a binarysearch is used for the solution τ_(*). In other words, a(τ_(new)), iscomputed using the middle point ##EQU19## and comparing τ_(new) witha(τ_(new)). If τ_(new) >a(τ_(new)), a search is conducted on the lowerhalf of [a₀, τ₀ ]; If τ_(new) <a(τ_(new)), the upper half is searched.One continues by computing a(τ) for the new middle points of the newsearch region until a stopping criterion is satisfied.

The terms to be used in a BS E-TAMI algorithm are defined as follows:

b: the bottom end of the search region.

t: the top end of the search region.

m: the middle point of the search region.

SSPDET (N,τ): SSP detector using a threshold τ where the maximumallowable number of SSPs is N.

N(τ): the number of SSPs detected by SSPDET (N,τ).

Note that N(τ)<N.

pos(τ): positions of SSPs detected by SSPDET (N,τ).

dτ: previous threshold m satisfying m>a(m).

dpos: previous positions corresponding to dτ.

The algorithm to find N positions is described generally as followsbelow: ##STR2##

In step 6, dτ becomes the desired solution because it is just before theposition where τ becomes larger than a(τ), which means τ is closest toa(τ). A more detailed description of the algorithm for BS E-TAMI isshown in FIG. 31, for steps 133 through 145.

If a brute force search for τ_(*) were used, the required computation ison the order of 10⁴ assuming τ₀ ≈10⁴ and a₀ ≈0. The required computationusing a binary search is [(log₂ L) +1] when the data size is L, whichbecomes about 15 when L=10⁴. About a thousand-fold computational savingis obtained using the binary search. When monotonicity fails due toperiodic motion for example, the heuristic is to use the initial SSPpositions, pos(τ₀).

This approach of employing a binary search can easily be combined withthe adaptive N scheme previously discussed. The advantage of this binarysearch approach is that it is fast and very simple compared to E-TAMI(equidistant TAMI). The disadvantage is that it fails when themonotonicity assumption is not satisfied. However, the assumption isvalid for most GOPs in ordinary video material.

Hardware System/Software Implementation:

The present inventors conceived a software implementation of the TAMIand OSA embodiments of the invention, as shown in FIGS. 32 through 38,which will now be described. Note that the hardware implementation forthe TAMI embodiment of the invention is shown in FIG. 3, as previouslydescribed.

In FIG. 32, a flowchart for a software implementation embodiment forboth the TAMI and OSA embodiments, as shown, includes steps 150 through161. Note that this software implementation is applicable for all TAMIembodiments, including general TAMI, VBR TAMI, and FBR TAMI. Morespecifically, after starting the software routine in step 150, scenechange detector step 151 is entered for detecting accumulated slowchanges or complete slow changes, whereby different versions forsoftware routines may be required for the various embodiments of theinvention. The next step 152 or software routine "I1 picture coding"provides for initialization of the group of pictures or GOP beingprocessed. The first frame is coded as an I frame in correspondence tothe MPEG standard. However, in the present invention, the first frame ismore specifically coded as an I1 picture providing full resolutioncoding, whereas as previously described, I2 coding may be used for otherframes within the GOP being processed in association with the detectionof Type 1 scene changes, whereby the I2 coding is coarser than the I1coding. The next step 153 determines when the current picture index oractual picture number (CPICI) corresponds to the first frame of a GOP.In this step the variable is set to be the first picture number (FPN).The next step 154 provides for an index "i" for a scene segment, whichindex is initially set to zero for the first scene segment, andsuccessively increased by one as additional scene segments or frames areprocessed.

The next step 155 processes the data through a encoding algorithm(MAIN), which includes some of the MPEG standard with the addition ofnew steps conceived by the inventor, as will be described below.

In the next step 156, the index for a scene segment "i" is incrementedby one for advancing the processing to the next scene segment.

The next step 157 is a decision step for determining whether the index"i" is less than the number of scene segments (SCNUM) or loops asdetermined by the scene change detector in step 151. Note that a highnumber of segments is assigned if there are a high number of scenechanges. In other words, the number of segments is directly proportionalto the number of scene changes detected, thereby making the system anadaptive system.

After the final scene segment has been processed through the loopincluding steps 155 through 157, decision step 157, after determiningthat the final scene segment has been processed, exits from theprocessing loop into step 158, in which the actual picture numbercorresponding to the first frame of the GOP just processed isincremented by 15. Note that in this example, as previously explained,the GOP size chosen is 15 in the preferred embodiments of the invention,but per the MPEG standard can be other than 15.

The next step 159, SCDET, detects the scene changes for the next 15frames, i.e. the frames of the next GOP to be processed.

The next step 160 determines whether the current picture index or numberCPICI, incremented by 15, is less than the last picture number LPN. Ifthe answer is yes, the data for the current frame being processed is fedto step 154, for continuing processing in the loop represented by steps154 through 160. Once the last picture number LPN is processed, decisionstep 160 exits from the loop, ending the encoding process in step 161.

In FIG. 33, the scene change detection processing, SCDET, for a FBRTAMIor fixed bit rate TAMI, is shown. This flowchart is similar to theflowchart of FIG. 4, but provides greater details of the processing.Note that the SCDET for VBRTAMI is shown and described above inassociation with FIG. 10. Similarly, the SCDET embodiment for the OSAembodiment of the invention is shown and described above in associationwith FIG. 11.

In FIG. 33, the SCDET for the FBRTAMI or fixed bit rate TAMI begins withthe loading of the GOP frames into memory in step 170. IN step 171counters in the system are initialized. In the example given for step172, the scene index for a scene change is shown as being set to 2, thepicture number of the first frame of a scene segment for a current frameis shown as set to 0, and the index for a scene segment is incrementedby 1 before proceeding to the next step. In step 173, scene segment dataf₀ is copied into a frame reference memory f_(ref), and a currentpicture frame counter is set to 1. Next, in decision step 174, adetermination is made as to whether the distance or movement between acurrent frame f_(c) and an immediately previous reference frame f_(c-1)is greater than a threshold T₁ for a Type 1 scene change. If the answeris affirmative, step 178 is entered. If the answer is negative, step 175is entered, for determining whether the motion between the current frameand a previous reference frame exceeds a Type 0 threshold T₀. Ifaffirmative, step 180 is entered, otherwise step 176 is entered.

If step 178 is entered from step 174, the scene change Type is set to 1,the picture number c of the first frame of a scene segment isidentified, and the last frame of a previous scene segment is identifiedby (c-1); and the index for the scene segment is incremented by 1, asshown. Next, in step 179, the current frame data f_(c) is copied to thereference frame data f_(ref). From step 179, step 176 is entered fordetermining whether the end of the present GOP being processed has beenreached, that is whether the 15th frame has been processed. If theanswer is no, step 177 is entered for incrementing the current frame by1 to proceed to the next frame, then the processing loops back into step174. However, if the answer is yes, meaning that the frames within theGOP have been processed, step 184 is entered for processing the nextGOP. Note that in FIG. 33, in the legend shown, that "D(,)" designates adistance measure for the amount of motion between frames. Also, thenotation "Cond. A:sct[scindex-1]=1 &(pNSCL[scindex-1]-pNSCF[scindex-1])=0", means that a scene change typeof a previous segment is 1, and its scene duration is 0.

FIGS. 34 and 34B show an "N-P TAMI SCDET" scene change detector for usein the TAMI encoder routine of FIG. 32. Note that a substantial numberof the steps in the flowchart of FIGS. 34A and 34B for N-P TAMI SCDETare the same as the steps shown in a portion of the flowchart of FIG.33, wherein the reference designations for these steps are identical.For example, the initialization steps 170 through 173 are the same forthe flowcharts of FIGS. 33, 34A, and 34B. Steps 200 through 205 of theN-P TAMI flowchart of FIGS. 34A and 34B are different than the SCDETFBRTAMI flowchart of FIG. 33. In the flowchart of FIGS. 34A and 34B,steps 174, 178, 201, and 202, detect a scene change of Type 1 whereassteps 180 through 183, and 200, detect a scene change of Type 0. Also,further note that these steps indicated for providing the Type 1 andType 0 scene changes, together with step 179, provide an Exclusive ORfunction. Also note that steps 203 through 205 provide for the insertionor generation of default positions for P1 designated frames. Forconvenience, in this example, such default positions are designated as ascene change Type 2.

The picture coding step 152 of the TAMI encoder flowchart shown in FIG.32 is illustrated in greater detail in the flowchart of FIG. 35. Steps250 through 256 provide for I1, I2, P1, P2, B1, and B2 coding, asillustrated. Step 250 provides for a discrete cosine transform of thedata. Step 251 provides the main data compression, and is a quantizerhaving step sizes that can be adapted to provide different quantizationlevels. Step 252 provides variable length coding VLC, such as Huffmancoding, for example. The buffer control provided in step 256 is includedas part of the MPEG standard bit rate control for making the output bitrate constant, whereby if excess bits are being used the quantizer iscontrolled for coarser quantization in order to reduce the bit use.Steps 253 and 254 provide inverse quantization, and inverse discretecosine transform processing, respectively. Step 255 provides for savingor storing in memory the decoded results from the inverse discretecosine transform 254, which results are used for later motioncompensation. Note that as shown in the legend, the present inventorselected to provide the quantizer in step 251 with 6 different default QSlevels, whereby the greater the value of QS, the less resolution orcoarser the quantization provided in step 251. As shown in this example,frames designated as I1 and P1 each have a designated quantization levelof QS. Frames designated as bidirectional frames B1 or B2, each havequantization levels of 2QS. Frames designated as I2 have quantizationlevels of 10QS, and frames designated as P2 have quantization levels of3QS.

The MAIN encoding algorithm shown as step 155 in FIG. 32 for the TAMIencoder is shown in detail in the flowchart of FIGS. 36A and 36B forsteps 260 through 278, and 280 through 285, respectively. Morespecifically, in step 260 a count of past P1 frames is kept, asdesignated by NP1, which initially has a "0" count as shown in theillustration. In step 261, a determination is made as to whether a scenechange of Type 1 is attained. If the answer is yes, step 262 is enteredfor equating the current picture number PN to the picture number of the"ith" frame of a scene segment. Next, a decisional step 263, is enteredfor determining whether the picture number represents the last frame ofthe GOP. If so, step 264 is entered for coding the frame as an I1 frame,if not, step 275 is entered for coding the frame as a I2 frame.

If in step 261, a scene change of Type 1 is not detected for the currentframe being processed, step 273 is then entered for determining whetherthe motion between frames for the frame being processed is a Type 0change, that is whether the Type 0 scene change has been attained. Ifnot, step 265 is entered. If so, step 274 is entered for decrementingthe first picture number of the current scene segment by 1, as shown.Note that the further processing steps for the I1 coding step of step264, and I2 coding step 275, is shown in the flowchart of FIG. 35,infra.

In step 265, it is determined whether the duration of the current scenesegment being is zero or not. If so, step 156 of the flowchart shown inFIG. 32 is entered. If not, step 266 is entered to set the picturenumber to the last frame number of the segment being processed. Next,decisional step 267 is performed for determining whether the frameposition is the last frame of the GOP. If yes, step 276 is entered forperforming telescopic motion estimation for all B frames between thefirst and last scene segments of the associated picture number,whereafter step 277 is entered for I1 coding via steps 250 through 256in the flowchart of FIG. 35. However, if in step 267 it is determinedthat the frame position is not the last frame position, step 268 isentered for performing motion estimation for a P frame and preceding Bframes via steps 290 through 297 of the flowchart of FIG. 37. After step297 of the flowchart of FIG. 37, step 269 is entered for predictionprocessing in accordance with the MPEG standard. Next, step 270, adecision step, is entered for determining whether the next scene has aType 1 scene change, and whether any P1 frames were previously detected.If the answer is no, P2 coding is conducted in step 278 via steps 250through 256 of FIG. 35. If the answer is yes, P1 coding is conducted viasteps 250 through 256 of FIG. 35. Next, step 272 is entered forincrementing by 1 the count for the number of past P1 frames.

Step 280 (FIG. 36B) is then entered for setting the current picturenumber to the picture number of the first frame in the last scenesegment incremented by 1. Next, interpolation step 281 is entered forconducting interpolation in accordance with the MPEG standard.Thereafter, step 282 is entered for determining whether the number of P1frames is equal to 0. If the answer is yes, the B2 coding step 283 isentered. If the answer is no, the B1 coding step 285 is entered. Notethat the substeps 250 through 256 for steps for 283 and 285 is shown inthe flowchart of FIG. 35, as previously described. Next, decisional step284 is performed for determining whether the current picture number PNis less than the picture number of the last frame of a scene segment"i". If the answer is yes, the processing loops back to step 281,whereas if the answer is no, the process proceeds to step 156 (see FIG.32) for incrementing by 1 the index "i" for a scene segment.

For the MAIN step 155 (see FIG. 32) or MAIN encoding algorithm, asdescribed for the flowchart of FIGS. 36A and 36B, the motion estimationstep 268 for determining the motion associated with P frames, is shownin detail in the flowchart of FIG. 37. As shown, the first step 290 isfor sending the current frame number to the picture number of the "i"scene segment incremented by 1. Next, in step 291, the forward motionvector search between the current frame and the picture number scenesegment of the last reference frame is computed. Next, the current framenumber is incremented by 1 in step 292. Thereafter, step 293 is enteredfor determining whether the current frame is less than the picturenumber of the last frame of a scene segment. If the answer is yes, step291 is entered through a loop, whereas if the answer is no, step 294 isentered for setting the current frame number to the picture number ofthe last scene segment decremented by 1. Next, step 295 is entered forconducting a backward motion estimation, using in this example thestandard algorithm from the MPEG standard. Next, step 296 is entered fordecrementing by 1 the current frame number FN. Thereafter, step 297 isentered for determining whether the current frame number is greater thanthe picture number for the first frame of the scene segment of the firstframe. If the answer is yes, the processing loops back to step 295,whereas if the answer is no, the processing proceeds then to theprediction step 269 shown in FIG. 36A.

In the flowchart of FIG. 36A for the MAIN encoding algorithm of the TAMIencoder of FIG. 32, the motion estimation step 276 for I frames is shownin detail in the flowchart of FIG. 38 for steps 300 through 307. Notethat the flowchart of FIG. 37 for the motion estimation steps for Pframes is almost identical to the flowchart of FIG. 38. In other words,steps 290 through 292 and 294 through 297, of the flowchart of FIG. 37are identical to steps 300 through 302, and 304 through 307,respectively, of the flowchart of FIG. 38. The only difference betweenthe MEP and the MEI processing is between steps 293 of the former and303 of the latter. Step 293 includes the last frame in the associateddetermination step, whereas step 303 excludes the last frame, andtherefore is only determinative of whether the current frame number isless than the last frame.

A hardware system for permitting various embodiments of the invention asdescribed above to be accomplished will now be described in greaterdetail than for the systems of FIGS. 3 and 9. With reference to FIG. 39,the system includes a scene change detector 310 for receiving videodata, and designating the various frames as I, P, or B, via detection ofscene changes of Type 0 and/or Type 1. The associated GOP is storedwithin the scene change detector 310. The Control Signal CS is obtainedfrom a microprocessor 334, programmed to carry out the required steps,as previously described. One output signal on line A1 is a picture modesignal (PMS), which is a control signal identifying the designation ofthe frame being processed as either an I, P, or B frame. This signal isconnected as an input signal to a motion compensator 318, which isdescribed in greater below. Another output signal along line A2, is aframe output signal The signal is connected to a summing junction 312,and to the motion compensation module 318 via G1. An output along lineG0 from motion compensator 318 is connected to the summing junction 312.The output signal or data of the summing junction 312 is connected to adiscrete cosine transform module 314, which is connected in cascade witha quantization module 316, variable length coding module 324,multiplexer module 328, bit counter module 330, and a buffer 332.Feedback lines J1 and J2 are connected from buffer 332 and bit counter330, respectively, to a bit rate controller 326, the output of which isconnected along line J0 to the quantizer module 316. Another output ofthe quantizer module 316 is connected to an inverse quantizer 320 (Q⁻¹),the output of the latter being connected as in input to an inversediscrete cosine transform module 322. The output of the inverse discretecosine transform (IDCT) module 322 is connected to another summingjunction 323, which junction also receives an output signal along G0 ofthe motion compensation module 318. The output from a summing junction323 is connected via G2 to an input of the motion compensation module318. Note that the portions of the encoder enclosed within a dashed linearea designated 308 are representative of a typical MPEG video encoderknown in the art. The present inventors added the scene change detectormodule 310, bit rate controller 326, and motion compensation module 318to the prior encoder, for modifying the same to accomplish the variousembodiments of the present invention. Note that the output line MV fromthe motion compensation module 318 to the multiplexer (MPX) 328 providesmotion vector bits to the latter. Also, the inverse quantizer 320 andinverse discrete cosine transform module 322 simulate a decoder, as isconventional in the art. The bit counter 330 serves to count bitsgenerated in the system for determining the output behavior of thesystem, in order to control the quantizer 316 in a manner ensuring thatthe number of bits being utilized do not exceed the capability of thesystem. The bit rate controller 326 adjusts the coarseness of quantizer316 to accomplish this result.

The scene change detector module 310 will now be described insubstantially greater detail with reference to FIGS. 40 through 45.Further reference is also made to the algorithms previously describedabove, particularly the algorithm of FIG. 32. As shown in FIG. 40, thescene change detector 310 includes a frame memory 336 connected to adistance computation unit 338, and a scene change detector controller346, as shown. The frame memory 336 is also responsive to a scene changedetector control signal (SCDCS). The frame memory 336 is a standardframe memory for 16 frames, in this example, assuming a GOP of 15frames. The distance computation unit 338 computes the distances ormotion between current and subsequent or following frames, and is alsoresponsive to the SCDCS signal. It is also responsive to a referenceframe number signal F_(ref) from a Type 0 scene change detector module342, or from a Type 1 scene change detector module 340, as shown. Notethat the feedback signal F_(ref) is a feedback signal that the distancecomputation unit responds to for resetting the reference frame positionsused in distance or motion computation between frames, as previouslydiscussed. The Type 1 scene change detector module 340 also providesoutput signals to both the Type 0 scene change detector 342, and to theGOP structure generation unit 344, as shown. The latter two scene changedetector modules 340 and 342 are discussed in greater detail below. TheGOP structure generation unit 344 is controlled via the SCDCS controlsignal, and provides an output along E3 to the scene change detectorcontroller 346, and receives a signal along E2 from the latter.Controller 346 also receives a signal via FO from the frame memorymodule 336, and provides a frame output signal on line or bus A2, andthe picture mode signal PMS on signal line A1. The GOP structuregeneration unit 344 detects positions used to generate the complete GOPstructure or map in accordance with the MPEG standard. Also, note thatthe SCD controller module 346 can be provided by a small microprocessoror by hardwired logic for the timing, data flow, synchronization, and soforth.

The distance computation unit 338 is shown in greater detail in FIG. 41.As shown, a histogram module 348 is included for providing a histogramof the previous reference frame from data received along B1, and anotherhistogram module 350 is included for producing a histogram of thecurrent frame, the data for which is received along line B0. Thehistogram module 348 is connected to a memory 352 for storing theprevious reference frame data until the memory 352 is reset by the scenechange detector module 340 or scene change detector module 342. Anoutput of memory 352 is connected to an absolute difference unit 354,which also receives the histogram data from histogram module 350 for thecurrent frame. The absolute difference therebetween is computed by unit354 and outputted to an adder module 356, the output of which is fedalong line B1 as the histogram difference to the scene change detectormodules 342 and 344. Note that although the example of FIG. 41 for thedistance computation unit 338 shows the use of Histogram modules 348 and350, block variance processing, histogram of difference processing, andso forth, could have alternatively been used in place of the histogramtechnique. These other techniques have been previously described above.

With reference to FIG. 42, the Type 1 scene change detector module 340will now be described. A comparator 358 is included for receiving thedistance or motion signal along line B2 from the distance computationunit 338, and comparing the signal with the Type 1 threshold signal T1,whereby if B2 is greater than T1, the output of the comparator isindicative of a digital "1", whereas if the signal B2 is less than T1,the output of the comparator 358 is representative of a digital "0". Theoutput from comparator 358 is fed to both a detection signal unit 360which acts as a buffer, and to a position output unit 362 which acts asa frame number generator for providing F_(ref), which is set to F_(c)(current frame number), along the line designated at a lower end as C2and at an upper end as B1, as shown. Note that the buffer 360 is anoninverting buffer. Accordingly, if a digital "1" is provided as aninput signal thereto, the output signal from the detection signalgeneration unit 360 will also be a digital "1".

The Type 0 scene change detector module 342 will now be described ingreater with reference to FIG. 43. As shown, the comparator 364 isincluded for comparing a distance or motion measurement signal D1 withthe T0 threshold signal, for producing a digital "1" output if D1 isgreater than T0, and a digital "0" if the distance D1 is less than T0threshold. Comparator 364 also receives a Type 1 detection signal alongD1 from the Type 1 scene change detector module 340, for inhibitingcomparator 364, for in turn inhibiting the Type 0 scene change detectormodule 342 if a Type 1 scene change is detected by module 340. Theoutput of comparator 364 is connected as an input to a position outputunit 366 which provides along B1 the F_(ref) signal which is equal toF.sub.(c-1) for the frame number of the previous frame. Also, theposition output unit 366 provides a signal along D3 indicative ofwhether a Type 0 scene change has been detected between frames.

The GOP structure generation unit 344 of FIG. 40 will now be describedin greater detail with reference to FIG. 44. As shown, unit 344 includestwo memories 368 and 370 for storing Type 0 scene change positions, andType 1 scene change positions, respectively. These memories 368, 370 areindividually connected to inputs of a reference frame positioning unit372, which determines I and P frame positions based upon the detectedscene changes. One output from the reference frame positioning unit isconnected to a B frame positioning unit 374, for designating remainingframe positions not designated as I or P frames, as B frames. Thereference frame positioning unit 372 is also connected to a PMODE memory378. The PMODE memory 378 also receives the B frame positions from unit374, and serves to store the I, P and B frame positions for the GOPbeing processed. In this example, the PMODE memory 378 contains 16registers designated as the 0 through 15 registers relative to frames "0to 15", respectively. The PMODE memory 378 receives along line E2 aPMODE memory control signal, and outputs along line E3 a PMODE readsignal.

The scene change detector controller module 346 of FIG. 40 will now bedescribed in greater detail with reference to FIG. 45. Controller 346includes a control circuit 380 that can include either a smallmicroprocessor or be provided by handwired logic. The control circuit380 is receptive of a PMODE read signal along line E3, and outputs botha PMODE memory control signal along line E2, relative to the GOPgeneration unit 344 (see FIG. 40). The control circuit also outputs thescene change detector control signal SCDCS. As further shown, controlcircuit 380 is connected to a picture mode signal (PMS) generation unit384, and to a frame sequencing unit 382. The frame sequencing unit 382acts as a buffer, and functions to receive frame data along line F0 fromframe memory 336, the data being representative of the actual image dataof the frame being processed, whereby the frame sequencing unit 382provides frame data as an output along line A2 representative of theframe being processed. Also, the picture mode signal generation unit 384provides along line A1 the picture mode signal (PMS) that represents aswitch control signal described in detail below, for permitting detailedidentification of the frame being processed.

In FIG. 46 the motion compensation module 318 shown in FIG. 39 is shownin detail. The components enclosed within a dashed line rectangular area385 represent a motion compensation circuit known in the prior art. Thepresent inventors added two modules to the known motion compensationnetwork, which modules include a telescopic motion estimator (ME) module396, and a switch control block module 408, as shown. The prior motionestimation network 385 includes one IMODE or intraframe mode switch 386,and four BMODE switches 388, 394, 400, and 402, respectively. Threeother data transfer switches 398, 410 and 412 are also included, asshown. Each one of these switches are controlled by an MSC or modeswitch control signal, as shown, which signal is generated by a switchcontrol block 408 in response to the picture mode signal PMS receivedalong line A1. The truth table for switch control block 408 is shown inFIG. 47 as Table 414. Also, as shown, each of the switches are operablein response to the MSC signal for switching their associated switch armbetween a "0" and "1" contacts. A B frame compensator 390 has individualconnections to switches 388, 394, 400, and 402. A P frame compensator392 has individual connections to switches 388, 394, and 398,respectively. Switch 394 has its switch arm connected to a motionestimator module 396. Switch 398 has its arm connected to the P framecompensator 392, and its "1" and "0" contacts connected to the "0"contact of switch 400, and "0" contact of switch 402, respectively. Theswitch arm of switch 400 is connected in common to a frame memory F0404, and via an H2 line to a motion estimator module 396. A switch armof switch 402 is connected in common to a frame memory F1 406, and via asignal line H3 to motion estimator module 396. Frame memories 404 and406 are also connected to the "1" contacts of switches 410 and 412,respectively. The switch arms of 410 and 412 are connected in commonalong line G2 to a summing junction 323 (see FIG. 39).

Operation of the motion compensation module 318 will now be describedwith reference to FIGS. 46 and 47. The B frame compensator module 390and P frame compensator module 392 are operative for providing thenecessary interpolation functions for processing B and P frames,respectively. When the PMS signal is "0, 1", the IMODE switch 386connects a 0 signal to summing junction 312. Switches 410 and 412operate in an alternate manner for connecting an input of either framememory F0 404 or frame memory F1 406 to the summing junction 323. Theseswitches are switched alternatively for alternately connecting memories404 and 406 to the summing junction 323 so long as the value of thepicture mode signal PMS is either 0, 1, 2, or 3. However, if the valueof the PMS signal is either 4 or 5, switches 410 and 412 remain in theirlast switch position so long as the value of the PMS signal remains ateither 4 or 5. If the value of the PMS signal is either 0 or 1, switch386 is operated for connecting 0 volt to summing junction 312 alongsignal line G0; B mode switch 388 is operated for disconnecting theoutput of the P frame compensator 392 from the circuit or network; Bmode switch 394 is operated for connecting the output of the motionestimator module 396 as an input to the P frame compensator 392; switch398 is operated for connecting the input of the P frame compensator 392either through switch 400 or switch 402 to frame memories 404 and 406,respectively, depending upon which one of these frame memories is themost current that has been loaded with data; and switches 400 and 402are operated for connecting the outputs of frame memories 404 and 406 tothe inputs of the B frame compensator 390. When the value of the PMSsignal changes to a "2" or "3", switch 386 and 388 are operated forconnecting the output of the P frame compensator 392 to the summingjunction 312, and switches 400 and 402 are operated for individuallyconnecting the outputs of memories 400 and 406 to switch 398, wherebythe latter operates to connect one of the frame memory outputs frommemories 404 and 406 to the input of the P frame compensator 392dependent upon the position of the switch arm of switch 398 at the time.

The motion estimator module 396 will now be described in greater detail.With reference to FIG. 48, motion estimator module 396 includes a motionvector memory 420 connected between a telescopic motion estimatorcontroller 424 and estimator module 426. A frame memory 422 is connectedto estimator 426, and telescopic motion estimator controller 424.Another frame memory 428 is connected to estimator module 426, and thecontroller 424. A third frame memory 430 is also connected to controller424 and estimator module 426. The motion vector memory 420 for storingmotion vectors for future and prior reference frames, and for B frameslocated between such reference frames. This memory is controlled andaccessed via the telescopic ME controller 424. Frame memory 422 is usedto store current frame data temporarily, until such data is required foruse. The estimator module 426 performs the actual motion vector search,and uses any conventional motion estimator method, including a fullsearch method, or any of the other prior methods as previously describedherein. The telescopic ME controller 424 controls the timing for theother modules of the associated motion estimator module 396, and readsthe motion vectors from the motion vector memory 420, and outputs theestimated motion vectors to the B frame compensator 390, or P framecompensator 392 via the H1 signal line. Frame memories 428 and 430 areused for storing reference frames, where at any given time one of thesememories will store the immediate reference frame, and the other ofthese memories will store the future reference frame. Frame data isbrought into the frame memory 428 via the H2 signal line, and frame datais brought into the frame memory 430 via the H3 signal line.

The telescopic motion estimator controller 424 of the telescopic motionestimator module 396 will now be described in greater detail withreference to FIG. 49. As shown, controller 424 includes a memory 432connected to a control circuit 436. The controller circuit 436 isconnected to two read only memories (ROMs) 434 and 438, respectively.ROM 434 is associated with a forward motion estimation (FME) sequence,whereas ROM 438 is associated with a backward motion estimation (BME)sequence. Control circuit 436 is operative for providing the estimatorcontrol signal along signal line 13, the motion vector memory controlsignal along signal line I1, and the frame memory control signal alongsignal I2, respectively.

The bit rate controller 326 will now be described in greater detail withreference to FIG. 50. As shown, controller 326 includes a quantizationparameter or QP adjustment module 440, and a target bit adjustmentmodule 442. A truth table 444 is shown, which indicates how the scalingfactor X for each picture type is changed in correspondence to the valueof the picture mode signal PMS received by the quantization parameteradjustment module 440. This module is programmed for computing theequation Q_(p) =31X(F/J). By using 31 steps in this quantizationparameter equation, five bits may be used to designate the same. Theratio of buffer fullness F to buffer size J lies between zero and one,depending upon how many bits are associated with buffer 332 (see FIG.39), i.e. from 0 to J, typically J represents the average bits needed tocode about five pictures or 250Kbytes, when the bit rate isapproximately 1.5 Mbyte/sec. Note with further reference to truth table444, that when the PMS has a value of "0", the scaling factor of 1 isrelative to a I1 frame. When the PMS signal is "1", the scaling factor Xis ten, in association with a I2 frame, which is a coarsely quantizedframe in this example. When the PMS signal has a value "2", the scalingfactor X is "1", in association with a P1 frame. When the PMS signal is"3", this scaling factor is 3, in association with a P2 frame, which isa somewhat coarsely quantized frame. When the PMS signal is "4", thescaling factor X is 2, in association with a B1 frame When the PMSsignal is "5", the scaling factor X is 2, in association with a B1frame. Lastly, when the PMS signal is "5", the scaling factor X is 2,for a B2 frame. Further note that the target bit adjustment module 442operates to compute the equation D_(t) =X_(t) T_(gop) /E_(gop), wherebythe legend in FIG. 50 defines each one of the components of the equationfor D_(t), i.e. the target bit allocation for picture Type t.

The embodiment of the invention for BS E-TAMI (Binary Search EquidistantTAMI), as presented above in association with the algorithm shown inFIG. 31, can be carried out using the general hardware configuration ofFIG. 39. However, the scene change detector or SCD 310 is configureddifferently than for other embodiments of the invention. A generalizedblock diagram of the required SCD 310 configuration is shown in FIG. 51.The frame memory 336, GS or GOP structure generation unit 344, and theSCD controller 346, are as previously described for the embodiment ofFIG. 40. The difference lies in the use of the binary search unit 450between the frame memory 336 and the GOP structure generation unit 344,as shown.

The configuration of the binary search unit 450 is shown in greaterdetail in FIG. 52. The threshold unit 452 operates to compute the valuefor m, the middle point of the search region of the frame beingsearched. Note that the frame data is carried over bus B0 from the framememory 336. Control unit 454 operates to provide the appropriate timingand binary search control signal BSCS. Control unit 454, in response tothe outputs of the scene change detectors 340 and 342, provides a signalindicative of a Type 0 scene change along output line D3, and the frameposition for the Type 0 scene change along line C2. The search regioncomputation unit 456 determines the next search region for conducting abinary search from one step to another of the associated algorithm,whereby the searching is conducted in an iterative manner, as shown inFIG. 31.

Subband Video Coding with TAMI:

It is known in the art to subsample a discrete-time signal as a step inseparating its lowpass and highpass frequency components. This techniquehas been extended to processing of an image or a frame of video byapplying subsampling along each of the two spatial directions. Whenfollowing appropriate highpass and lowpass filters, subsampling by afactor of two in each spatial direction decomposes an image or videoframe into four subimages, each of which has one quarter as many pixelsas the original image. These subimages (subband images) may be labelledas (L_(v), L_(h)), (L_(v), H_(h)), (H_(v), L_(h)), and (H_(v), H_(h))where the uppercase letters denote the type of filter (H=highpass,L=lowpass) and the subscripts denote the spatial processing direction(v=vertical, h=horizontal). The subband images may be recombined byinterpolation and filtering to reconstruct the original image from whichthey are derived. Any or all of the subband images may be furtherprocessed by the same steps to produce subband images of smaller sizes.In transmission or storage system applications, frequency selectiveimage coding may be achieved by using different coding methods ondifferent subband images.

In yet another embodiment of the invention, a new subband video codingalgorithm with temporally adaptive motion interpolation was conceived.In this embodiment, the reference frames for motion estimation areadaptively selected using temporal segmentation in the lowest spatialsubband of a video signal. Variable target bit allocation for eachpicture type in a group of pictures is used to allow use of a variablenumber of reference frames with the constraint of constant output bitrate. Blockwise DPCM, PCM, and run-length coding combined with truncatedHuffman coding are used to encode the quantized data in the subbands. Asshown below, the thresholds for the Type 1 and Type 0 scene changes areadjusted in direct proportion to the number of pixels subsampled in aframe. Simulation results of the adaptive scheme compare favorably withthose of a non-adaptive scheme.

Subband coding is known as having an inherent hierarchical resolutionstructure, which is suitable for prioritized packet video in ATMnetworks. Another known approach for video coding is motion compensationwhich has been recently standardized into the MPEG standard. Temporalredundancy reduction methods using subbands may be classified into twoapproaches. One is 3D spatio-temporal subband coding, and the other ismotion compensated 2D subband coding. The present subband embodimentapplies the latter approach in implementing the present fixed output bitrate subband video encoder shown in FIG. 53, and it provides improvedperformance in removing temporal redundancy due to motion.

In the known motion compensated 2D subband coding system of Y. Q. Zhangand S. Zafar, described in their paper "Motion Compensated WaveletTransform Coding for Color Video Compression", IEEE Trans. CircuitsSyst. Video Technol., Vol. 2, No. 3, pp. 285-296, Sept. 1992, forpurposes of determining motion vectors, two stages of subbanddecomposition are used, whereby the (L_(v), L_(h)) is decomposed by asecond stage of low pass filtering and subsampling. This produces a lowfrequency subband image with one-sixteenth the number of pixels as theoriginal image, comprised of the lowest one-fourth of the horizontal andvertical spatial frequency components of the original image, theso-called "lowest subband" image.

In the present subband embodiment, each picture of the input video isdecomposed into subbands by using biorthogonal filters. The motioncompensation scheme uses temporally adaptive motion interpolation(TAMI), as previously described. The number of reference frames and theintervals between them are adjusted according to the temporal variationof the input video.

More specifically, the algorithm for the present subband embodiment is aslightly modified version of TAMI as described above. It is modified toallow a variable number of P frames in each GOP. The new subband TAMIalgorithm takes the following steps for each GOP (group of pictures):

(1) It detects the positions of scene change of Type 1;

(2) It detects the positions of scene change of Type 0; and

(3) It determines the positions of all I, P, B frames.

For this embodiment, I, P, and B frames with full bit allocation aredenoted as I1, P1, and B frames, an I frame with reduced bit allocationas I2, a P frame with reduced bit allocation as P2, as with otherembodiments of the invention.

Two types of scene detectors are required in the algorithm, aspreviously described for other embodiments of the invention. The firstdetector declares a scene change of Type 1 for the current frame when adistance measure between the current frame f_(n) and the immediate pastfrom f_(n-1) is above a threshold T₁. This type of scene changecorresponds to an actual scene content change; it is coded as an I2frame (very coarsely quantized intra frame), and the immediate pastframe f_(n-1) is coded as a P2 frame (very coarsely quantized predictedframe). The I2 frame coding exploits the forward temporal maskingeffect, and the P2 frame coding takes advantages of the backwardtemporal masking effect. The second detector declares a scene change ofType 0 for the current frame when the distance measure between thecurrent frame and the last reference frame is above a threshold T₀. Inthis case the immediate past frame f_(n-1) becomes a P1 frame.

As indicated above, reference frame assignment strategy using Type 0scene change detection is that the end frame of every temporal segmentdetermined by a Type 0 scene change is a P1 frame, and that the framesin between should be B frames. Examples of GOP structures generated bythe TAMI algorithm are as previously shown in FIG. 8A and FIG. 8B.

In FIG. 53 a block diagram for the subband coding system 460 using TAMIis shown. The TAMI algorithm using multi-resolution motion estimation isapplied via SCD 310 and motion estimator 462 on the lowest of thespatial subbands after subband decomposition via the subband analysismodule 464. The motion compensated data are then quantized via quantizer316 and encoded by variable length coding in VLC module 324 using aHuffman code scheme, for example. The buffer 332 is necessary to convertthe output encoded data from variable rate to constant channel bit rate.

Two stages of subband filtering are used to generate seven spatial bands"1 through 7" as shown in FIG. 54. In this example, the filters areseparable 2D filters that use low-pass and high-pass biorthogonalfilters as taught by D. LeGall and A. Tabatabai in their paper entitled"Subband Coding of Digital Images using Symmetric Kernel Filters andArithmetic Coding Techniques" (Proc. ICASSP 88, pages 761-763, April1988), for subband analysis:

    H.sub.1 (z)=1/8(-1+2z.sup.-1 +6z.sup.-2 +2z.sup.-3 -z.sup.-4) (32)

    H.sub.h (z)=1/2(1-2z.sup.-1 +z.sup.-2)                     (33)

The constant factor, 1/8, for the low-pass filter of equation (32) waschosen to provide a DC gain of 1. The corresponding synthesis low-passand high-pass filters are G₁ (z)=H_(h) (-z) and G_(h) (z)=-H₁ (-z).These pairs of filters each have a perfect reconstruction property withthree samples delay. In other words, it is easy to show

    X(z)=z.sup.-3 X(z)                                         (34)

where X(z) is the input and X(z) is the reconstructed signal.

The temporal segmentation (scene change detection) algorithm is appliedon the lowest of the subbands, so that the amount of computation isreduced by factor of sixteen due to the reduced picture size of thelowest band.

The multi-resolution approach for motion estimation provided by MRME462, will now be described. The resolution level is set to s, whichcorresponds to the subband filtering stage. Let the maximum filteringstage be denoted by S, which is 2 in FIG. 54. In FIG. 54, s=2 for bands(1, 2, 3, 4) and s=1 for the others. The initial motion vectors areestimated only in band 1, and they are scaled to generate motion vectorsfor other subbands, as follows:

    d.sub.i (x,y)=d.sub.s (x,y)2.sup.S-2 +Δ.sub.i (x,y)  (35)

where d_(i) (x,y) is the motion vector at block position (x,y) inresolution level i, d_(s) (x,y) is the initial motion vector, and Δ_(i)(x,y) is the correction motion vector found by reduced search area. Theinitial motion vector, d_(s), is estimated by a full search with searchrange 4×4, where the block size is also 4×4. In a computerizedsimulation, the inventors set Δ_(i) (x,y) =(0,0) because the overheadbits for the correction usually exceeded the saving of data bits.

To allow a variable number of P1 reference frames in a GOP, a variabletarget bit allocation scheme is updated at the beginning of each GOP, asdescribed above for the adaptive selection of reference frames. Hence,the formula for target bit allocation is the same as equation "(30)"given above.

Within a GOP, the target bit allocation for each picture type is alsoallowed to vary to be adaptive to the changing scene complexity of theactual video sequence. The number of bits generated for the previouspicture having the same picture type is used as the target bitallocation. When the number of bits produced for one frame deviates fromthe target number of bits, the bit allocation for the next picture isadjusted to maintain an acceptable range of bit rate according to theequation: ##EQU20## where t is a picture type, with t ε{I1, I2, P1, P2,B}, D_(t) is target bit allocation for picture type t, X_(t) is thenumber of generated bits for the previous frame of the type t, EGOP isthe expected GOP bit rate computed by the most recent data of bitsgenerated for each frame type, and TGOP is the target GOP bit rate.T_(GOP) is computed by M(R/30), where M is the GOP size and R is thetarget bit rate (bits/sec). E_(GOP) can be computed by the equation:##EQU21## where A_(GOP) is the set of all picture types used in thecurrent GOP, n_(t) is the number of the frames of picture type t in theGOP, and X_(t) is either the generated bits for the previous frame ofthe type t or the initial bit allocation for picture type t when thepicture is at the beginning of a GOP.

There are two other concerns for bit rate control; one is to adjustactual coding bits to target bit allocation, and the other is the buffer332 content control to prevent a buffer overflow or underflow problem.Both of these control problems are handled by the following algorithm.At the end of each slice, the buffer 332 content is updated byF=F+S_(gen) -S_(t), where F is the buffer 332 content and S_(gen) is thenumber of bits generated for the slice, and S_(t) is the number oftarget bits for the slice. To maintain stable buffer behavior andregulate the generated bit stream to be close to the target bitallocation per frame, the quantization parameter, Q_(p), is adjustedaccording to the buffer fullness by using the relation:

    Q.sub.p =31×F/J                                      (38)

where J is the buffer size, taken as the amount of raw data in aboutthree to five pictures, and the number 31 means there are 31 differentnonzero step sizes which are coded by 5 bits.

Blockwise DPCM and a uniform quantizer are used only for the subband 1of an I frame. In the DPCM, horizontal prediction from the left is usedexcept for the first column where vertical prediction from above is usedas in FIG. 55. All subbands except the lowest subband of an I frame,regardless of the picture type, are coded by PCM (pulse code modulation)with a deadzone quantizer because there is little spatial correlation inhigh-pass filtered subbands as well as motion compensated residualimages. Horizontal and vertical scan modes are chosen according to thelow-pass filtered direction. Hence, the horizontal scan mode in FIG. 56is used for bands 2 and 5 (see FIG. 54). This mode is also used forbands 4 and 7 (see FIG. 54) by default because the bands are high-passfiltered in both directions. The vertical mode of FIG. 57 is used forbands 3 and 6 (see FIG. 54).

These scan mode selections contribute to statistical redundancyreduction by run-length encoding combined with Huffman coding. TheHuffman coding table is generated from run-length histogram dataobtained from several training image sequences including Tennis,Football, and flowergarden. Truncated Huffman coding is actually used tolimit codeword length. The codeword entries having length larger than 17bits, are replaced by fixed length codewords of either 20 or 28 bitswhich are defined in the MPEG standard.

Simulations were carried out using the Tennis and Football sequences tocompare the TAMI algorithm to a fixed scheme. Block type decisions for Pand B frames as in MPEG, 4×4 block size for s=1, 8×8 block size for s=2,and telescopic searching having half-pixel accuracy for motionestimation are used. As for the two temporal segmentation algorithms,difference of histograms of gray levels were selected for the distancemeasure. The thresholds used are 0.25N_(pix) for Type 1 detection,0.1N_(pix) for Type 0 detection, where N_(pix) is the number of pixelsin a single frame. All three color components (Y, U, and V) are encoded,and the bit rate results are computed by summing the compressed databits for the three color components, the bits for quantizationparameter, the motion vector bits, and the header bits, but the SNRcurves are for Y components only.

FIG. 58 shows a table of performance comparisons of average SNR andbits. It shows that TAMI is better than a nonadaptive scheme by 0.9 dBfor Tennis and 0.7 dB for Football. Although the SNR difference betweenthe two schemes has been shown to be slight, TAMI has a more stablepicture quality than the fixed scheme. TAMI automatically inserts more Pframes by detecting the global motion, such as zooming, for example.

In a real time display of the reconstructed sequences of Tennis andFootball, the quality differences between TAMI and a nonadaptive schemewere much more noticeable. The quality of TAMI was shown to be clearerand to have less blinking than that of the fixed scheme.

It was shown by experiments that adaptive selection of reference framesfor subband motion compensation compares favorably with the nonadaptiveschemes in terms of both subjective quality and an objective measure,SNR. The trade-off is that it requires a certain amount of encodingdelay because it needs to look ahead at GOP frames prior to encoding.The present embodiment provides a good scene adaptive scheme forremoving temporal redundancy using motion compensation in subbanddomain.

In FIG. 59, a block schematic diagram supplementary to and an expansionof FIG. 53 as shown, for giving greater details of a system for thesubband video encoding embodiments of the invention. The system as shownis very similar to the encoding system shown in FIG. 39, and likecomponents relative to the latter are shown in FIG. 59 with samereference designation, which components provide the same functions asdescribed for FIG. 39. The differences are that the motion compensationmodule 318 on FIG. 39 is replaced by the multi-resolution motionestimation module 462. Such a multi-resolution motion estimation module462 is known in the art, and is described in the paper of Y. Q. Zhangand S. Zafar, Ibid. Also, the discrete cosine transform module 314, andinverse discrete cosine transform module 322 of FIG. 39 are not includedin the system of FIG. 59. Another difference is that a scan mode switch321 is included between the quantizer 316 and the variable length codingmodule 324 in FIG. 59. The purpose of the scan mode switch 321 is toswitch between the various scan modes, for processing each subband.Lastly, another difference is that the subband analysis module 464 isincluded before the scene change detector 310 in the system of FIG. 59,and is not included in the system of FIG. 39.

The subband analysis module 464 of FIGS. 53 and 59 is shown in greaterdetail in FIG. 60. As shown, the subband analysis module 464 includes atiming controller module 526, responsive to a control signal CS foroutputting a timing control signal TC that is used to control all of thevarious modules of the remainder of the system shown. Video datareceived along line K0 is passed through a horizontal highpass filter500, and a horizontal lowpass filter 502. The output of horizontalhighpass filter 500 is passed through a subsampling module or decimatoror downsampler 505, for removing every other data sample, which areredundant after the filtering step, as is known in the art. Thesubsampled filtered video data is then passed through a verticalhighpass filter 504 and vertical lowpass filter 506, respectively, therespective outputs of which are passed through subsampling modules 505,for providing the filtered subband data along output lines labelled"Band 7" and "Band 6", respectively.

The filtered video data from the horizontal lowpass filter 502 is passedthrough a subsampling module 505, and provided as input data to both avertical highpass filter 508 and vertical lowpass filter 510. Thefiltered video output data from the vertical highpass filter 508 ispassed through a subsampling module 505, and outputted as video subbanddata along an output line labeled "Band 5".

The filtered output data from the vertical lowpass filter 510 is passedthrough a subsampling module 505, and delivered therefrom as input videodata to both a horizontal highpass filter 514, and a horizontal lowpassfilter 516. The filtered video output data from the horizontal highpassfilter 514 is passed through a subsampling module 505, and provided asfiltered video input data to both a vertical highpass filter 518 andvertical lowpass filter 520, the respective outputs which are passedthrough respective subsampling modules 505 and passed along subbandlines shown as "Band 4" and "Band 3", respectively.

The filtered video output data from horizontal lowpass filter 516 ispassed through a subsampling module 505, and therefrom provided asfilter input video data to both a vertical highpass filter 522 andvertical lowpass filter 524, respectively; the respective outputs ofwhich are passed through subsampling modules 505, respectively, andtherefrom outputted onto subband lines "Band 2" and "Band 1",respectively. All of the aforesaid horizontal and vertical highpass andlowpass filters enclosed within the dashed line area designated 499represent a known subband filtering system, as previously indicated, forproviding double filtering. The present inventors added a multiplexer512 for receiving the subband data from subband lines "Band 1" through"Band 7", respectively. The multiplexed output data is then providedalong output line A0 from multiplexer 512 for connection as input datato the scene change detector 310, as shown in FIG. 59.

Although various embodiments of the invention have been shown anddescribed herein, they are not meant to be limiting. Those of skill inthe art may recognize certain modifications to these embodiments, whichmodifications are meant to be covered by the spirit and scope of theappended claims.

What is claimed is:
 1. A method for compressing video data comprisingthe steps of:determining the degree of global motion between frames ofvideo data, where global motion is the motion between frames as a whole;assigning reference frames based upon the global motion; adjusting thetemporal spacing between reference frames relative to the degree ofglobal motion measured between frames relative to temporal masking inhuman vision, thereby determining the group of pictures (GOP) structure;establishing different threshold magnitudes or levels of motion betweenframes as representing different scene change types; assigning differentbit rates to individual frames based upon said pre-established thresholdlevels of motion between frames; said threshold establishing stepincluding the steps of designating a Type 1 scene change between a pairof successive frames as occurring whenever the measured motiontherebetween exceeds a value defining a T₁ threshold representing asubstantial scene or picture change; and designating and definingpursuant to the value of measured motion defining a Type 1 scene change,the first occurring or prior frame of the pair as a P2 frame, and thesecond occurring or past frame as an I2 frame, each being ofpredetermined bit rates via said assigning step.
 2. The method of claim1, further including the step of:assigning a bit rate for video codingeach frame based upon the degree of global motion measured betweenframes.
 3. The method of claim 2, further including the stepof:assigning reference frames as intra (I) and predictive (P) based uponthe degree of global motion measured between frames.
 4. The method ofclaim 3, further including the step of: assigning bidirectional (B)interpolated frames between reference frames.
 5. The method of claim 1,further including the steps of:said threshold establishing stepincluding a step of designating and defining a Type 0 scene changebetween frames occurring whenever the measured motion therebetweenexceeds a value defined as a T₀ threshold, whereby the T₀ thresholdrepresents substantial motion in a scene or picture; and detecting whenthe cumulative motion from an immediately preceding reference frame anda successive frame exceeds a T₀ threshold, for designating and definingthe immediately prior frame to said successive frame as a P1 frame ofpredetermined bit rate via said assigning step, said P1 frame being areference frame.
 6. The method of claim 1, wherein said assigning stepfurther assigns bit rates to said frames by using forward temporalmasking in human vision, whereby at a scene change an immediatelyfollowing frame is coarsely coded.
 7. The method of claim 1, whereinsaid assigning step further assigns bit rates to said frames in a mannerutilizing a backward temporal masking effect in a resulting codingscheme, whereby in instances where a scene change involves relativelylarge motion between two successive frames, the immediately past frameat the scene change and the immediately following frame are coarselycoded via the assignment of a relatively low bit rate.
 8. The method ofclaim 1, wherein said global motion determining step uses any one offive different distance measures for temporal segmentation includingdifference of histograms (DOH), histograms of difference (HOD), blockhistogram difference (BH), block variance difference (BV), and motioncompensation error (MCE).
 9. The method of claim 1, wherein saidassigning step includes the step of designating individual frames asintra frames (I), and/or predicted frames (P) based upon the magnitudeof motion between associated frames exceeding a given threshold level.10. The method of claim 1, further including the steps of:said thresholdestablishing step including a step of designating a Type 0 scene changebetween frames occurring whenever the measured motion therebetweenexceeds a value defined as a T₀ threshold representing substantialmotion in a scene or picture; and detecting when the cumulative motionfrom an immediately preceding reference frame and a successive frameexceeds a T₀ threshold, for designating and defining the immediatelyprior frame to said successive frame as a P1 frame of predetermined bitrate via said assigning step, said P1 frame being a reference frame. 11.The method of claim 10, further including the steps of:designatingsuccessive frames between reference frames as bidirectional B frames,respectively.
 12. The method of claim 10, wherein the magnitude of saidT₁ threshold is made about four times greater than the magnitude of saidT₀ threshold, whereby the T₁ threshold represents a complete scene orpicture change between a pair of successive frames.
 13. The method ofclaim 10, further including the step of:optimally spacing said P1designated frames for substantially optimizing the frame spacing betweenI1, I2, P1, and P2 reference frames.
 14. The method of claim 13, whereinsaid optimally spacing step includes the step of conducting a binarysearch between frames to find substantially equidistant positions forType 0 threshold change P1 designated frames.
 15. The method of claim13, wherein said optimal spacing step includes the steps of:minimizingthe deviation from the mean of distances between frame positionsinitially designated P1 through Type 0 threshold detection, and pairedframe positions designated I2 and P2, respectively, via Type 1 thresholddetection; and determining the structure for each GOP based upon theoptimally spaced P1 frames.
 16. The method of claim 15, wherein saidoptimal spacing step includes the steps of exhaustively searching eachframe for determining motion vectors.
 17. The method of claim 15,wherein said optimal spacing step includes the steps of using a backwardtelescopic search for determining motion vector information betweenframes.
 18. The method of claim 10, further including the stepsof:establishing a predetermined number of successive frames as a groupof pictures (GOP) thereby grouping said successive frames into aplurality of successive GOP's; and designating the first frame of eachof said plurality of GOP's as an I1 frame of predetermined bit rate viasaid assigning step.
 19. The method of claim 18, further including thesteps of:designating and defining frames as B1 frames between I₁ and P₁reference frames in each of said plurality of GOP's where at least oneType 0 scene change has been detected, said B1 frames each having apredetermined bit rate via said assigning step; and designating anddefining frames as B2 frames between reference frames in each of saidplurality of GOP's where no Type 0 scene changes have been detected,said B2 frames each having a predetermined bit rate via said assigningstep.
 20. The method of claim 19, wherein said I1 and I2 frames areintra frames, said P1 and P2 frames, are predicted frames, and said B1and B2 frames are bidirectionally interpolated frames.
 21. The method ofclaim 19, further including the step of gradually increasing to a fullbit rate the bit rate of frames following a scene change to render thedegradation of frames following a scene change imperceptible.
 22. Themethod of claim 19, further including the step of computing the bitallocations for B2 frames to follow the equation: ##EQU22## where N isthe number of |P1| frames, M is the number of frames in each GOP, |B1|is the bit allocation for B1 frames, and |P1| is the bit allocation forP1 frames.
 23. The method of claim 19, wherein said assigning stepfurther includes the steps of:assigning the relatively highest bit rateto I1 designated frames; assigning the second highest bit rate to P1designated frames; assigning the third highest bit rate to B2 designatedframes; and assigning relatively lower bit rates to I2, P2, and B1designated frames.
 24. The method of claim 23, wherein said assigningstep includes assigning bit rates of 200kB/sec. for I1 frames,100kB/sec. for P1 frames, greater than 10kB/sec. for B2 frames, and10kB/sec. for B1, I2, and P2 frames, respectively.
 25. The method ofclaim 23, further including the step of:allocating the same number ofbits from GOP to GOP for I1, I2, P1, and P2 designated frames,respectively.
 26. The method of claim 23, further including the stepsof:varying the bit allocations for each I1, I2, P1, P2, and B1 frametypes from GOP to GOP; and retaining the bit rate constant for each ofsaid frame types.
 27. The method of claim 26, further including in saidbit allocation varying step the step of assigning target bit allocationsfor each GOP in accordance with the following formula: ##EQU23## whereD_(t) is the target bit allocation for picture Type t, t being eitherI1, I2, P1, P2, or B1; C_(I1), C_(P1), and C_(B1), respectively; N isthe number of P1 frames, M is the GOP size or number of frames in a GOP;R=2G where G=[C_(I1) +NC_(P1) +(M-N-1)C_(B1) ]X; and X is a commonfactor determined from R and G.
 28. The method of claim 23, furtherincluding the step of:varying the number of bits allocated for I1, I2,P1, P2, B1, and B2 designated frames, respectively, from GOP to GOP inaccordance with the number of detected Type 0 scene changes; andmaintaining the ratio of bit allocations between I1, I2, P1, P2, B1, andB2 designated frames constant from frame to frame within a GOP.
 29. Themethod of claim 28, further including the step of:inserting extra P1frames into temporally busy regions, for producing a more constantperceptual picture quality.
 30. The method of claim 29, wherein saidinserting step further includes the step of satisfying the followingequation: ##EQU24## where k is the number of P1 designated frames, M isthe GOP size, R_(i1), R_(p1), and R_(b1) are bit rate allocations forI1, P1, and B1 frames, respectively, and R is the channel bit rate persecond, whereby R_(i1), R_(p1), and R_(b1) are adjusted to keep R at adesired constant value.
 31. The method of claim 23, further includingthe step of using the designated positions of P1 and/or P2, and B1,and/or B2 frames, for processing the frames through a motion compensatedinterpolation encoder, for encoding or compressing the associated videodata.
 32. The method of claim 31, further including the step of using atelescopic motion vector searching for determining motion betweenframes.
 33. The method of claim 31, further including the stepsof:sensing the bit rate being employed in said encoder in processingframes; and adjusting the coarseness of quantizing frames in saidencoder in proportion to the deviation or error between an actual bitrate and a target bit rate.
 34. The method of claim 33, furtherincluding the step of limiting the number of quantized P1 frames inhigh-motion segments between frames by replacing such P1 frames with B1frames for substantially maximizing the quality of the resultantpicture.
 35. The method of claim 33, wherein said adjusting step is inaccordance with the following equation: ##EQU25## where TB is target bitallocation, XTB is the target bit allocation for the previous frame,ABR_(GOP) is the actual GOP bit rate, and TBR_(GOP) is the target GOPbit rate.
 36. The method of claim 31, further including the stepof:inserting N P1 frames into the structure of each GOP by default, forreducing encoding delays and distances between reference frames, where Nis an integer number 1, 2, 3 . . . ; and processing each GOP throughsaid encoder via encoding the first frame through successive frames tothe first P1 default, and through the next successive frames to the nextoccurring P1 default, in an iterative manner until all frames have beenencoded.
 37. The method of claim 36, further including the stepof:sizing each GOP to have an even number of frames whenever N is odd;and sizing each GOP to have an odd number of frames whenever N is even.38. A system for compressing video data comprising video data associatedwith groups of pictures (GOP) including a predetermined number offrames, said system comprising:motion detection means for determiningthe degree of global motion between said frames, where global motion isthe motion between frames as a whole, wherein said motion detectionmeans includes Type 0 scene change detector means for detecting when thecumulative motion from an immediately preceding reference frame, and asuccessive frame exceeds a predetermined value defined as a T₀threshold, whereby said designating means responds by designating anddefining the immediately prior frame to said successive frame as a P1frame; means responsive to said global motion measurements from saidmotion detection means for both designating certain of said frames asreference frames, and adjusting the temporal spacing between referenceframes, thereby determining the GOP structure, wherein said designatingmeans includes means for coding said reference frames as I and/or P, andB frames relative to global motion between frames; and encoder means forencoding said reference frames.
 39. The system of claim 38, wherein saidmotion detection means includes:Type 1 scene change detector means fordetecting when the measured level of global motion between twosuccessive frames exceeds a predetermined value defined as a T₁threshold representing a substantial scene or picture change, wherebysaid designating means responds by designating and defining the firstoccurring of the two successive frames as a P2 frame, and the other orsecond occurring of the two successive frames as an I2 frame.
 40. Thesystem of claim 38, wherein said motion detection means furtherincludes:Type 1 scene change detector means for detecting when theglobal motion between two successive frames exceeds a predetermined T₁threshold representing a substantial scene or picture change, wherebysaid designating means responds by designating the first occurring ofthe two successive frames as a P2 frame, and the other or secondoccurring of the two successive frames as an I2 frame.
 41. The system ofclaim 40, wherein said frames are arranged in groups of pictures (GOP)each consisting of a predetermined number of successive frames, and saidencoder means further includes:bit rate controller means for insuringthat the number of bits being utilized in encoding a given GOP do notexceed the bit capability of said system.
 42. A method for compressingvideo data comprising the steps of:determining the degree of globalmotion between frames of video data, where global motion is the motionbetween frames as a whole; assigning reference frames based upon theglobal motion; adjusting the temporal spacing between reference framesrelative to the degree of global motion measured between frames relativeto temporal masking in human vision, thereby determining the group ofpictures (GOP) structure; establishing different threshold magnitudes orlevels of motion between frames as representing different scene changetypes; and assigning different bit rates to individual frames based uponsaid pre-established threshold levels of motion between frames, whereinsaid assigning step further assigns bit rates to said frames in a mannerutilizing a backward temporal masking effect in a resulting codingscheme, whereby in instances where a scene change involves relativelylarge motion between two successive frames, the immediately past frameat the scene change and the immediately following frame are coarselycoded via the assignment of a relatively low bit rate.
 43. A system forcompressing video data comprising video data associated with groups ofpictures (GOP) including a predetermined number of frames, said systemcomprising:motion detection means for determining the degree of globalmotion between said frames, where global motion is the motion betweenframes as a whole, wherein said motion detection means includes Type 1scene change detector means for detecting when the measured level ofglobal motion between two successive frames exceeds a predeterminedvalue defined as a T₁ threshold representing a substantial scene orpicture change, whereby said designating means responds by designatingand defining the first occurring of the two successive frames as a P2frame, and the other or second occurring of the two successive frames asan I2 frame; means responsive to said global motion measurements fromsaid motion detection means for both designating certain of said framesas reference frames, and adjusting the temporal spacing betweenreference frames, thereby determining the GOP structure; and encodermeans for encoding said reference frames.
 44. A method for compressingvideo data comprising the steps of:determining the degree of globalmotion between frames of video data, where global motion is the motionbetween frames as a whole; assigning reference frames based upon theglobal motion; adjusting the temporal spacing between reference framesrelative to the degree of global motion measured between frames relativeto temporal masking in human vision, thereby determining the group ofpictures (GOP) structure; establishing different threshold magnitudes orlevels of motion between frames as representing different scene changetypes; and assigning different bit rates to individual frames based uponsaid pre-established threshold levels of motion between frames, whereinsaid assigning step further assigns bit rates to said frames by usingforward temporal masking in human vision, whereby at a scene change animmediately following frame is coarsely coded.